[pgpool-hackers: 4233] Re: Issue with failover_require_consensus

[Tatsuo Ishii]

>> Hi Ishii-San
>> 
>> Sorry for the delayed response.
> 
> No problem.
> 
>> With the attached fix I guess the failover objects will linger on forever
>> in case of a false alarm by a health check or small glitch.
> 
> That's not good.
> 
>> One way to get around the issue could be to compute
>> FAILOVER_COMMAND_FINISH_TIMEOUT based on the maximum value
>> of health_check_peroid across the cluster.
>> something like: failover_command_finish_timouut = max(health_check_period)
>> * 2 = 60

After thinking more, I think we need to take account
health_check_max_retries and health_check_retry_delay as
well. i.e. instead of max(health_check_period), something like:
max(health_check_period + (health_check_retry_delay *
health_check_max_retries)).

What do you think?

> This is much better than my previous proposal.
> 
>> If you agree with the proposal I can cook up the patch and share it with
>> you.
> 
> I agree with you. Please go ahead.
> 
>> Thanks
>> Best regards
>> Muhammad Usama
>> 
>> On Mon, Nov 21, 2022 at 3:38 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>> 
>>> > Hi Usama,
>>> >
>>> > I think I found an issue with failover_require_consensus. When this
>>> > parameter is enabled, watchdog asks other watchdog to confirm the
>>> > failover event. If other watchdog replies back and the originator
>>> > watchdog can make a consensus, failover process begins. However, if no
>>> > replies arrive before FAILOVER_COMMAND_FINISH_TIMEOUT expires, the
>>> > failover request is discarded and failover will not begin. If this
>>> > only happens once or twice, we could expect that subsequent health
>>> > check would trigger failover. But actually this could repeat forever
>>> > (which means failover never happens) if health_check_period is larger
>>> > than FAILOVER_COMMAND_FINISH_TIMEOUT (currently 15 seconds). For
>>> > example, if health_check_period = 30 seconds, and other watchdog node
>>> > 1 starts 50 seconds after watchdog node 0 (suppose this is the leader
>>> > node), then every time failover consensus request is made (suppose the
>>> > time is t), it will be canceled at t + 15, because failover on
>>> > watchdog node 1 will happen at time t + 20 ( = 50 - 30).
>>> >
>>> > Since we allow other watchdog node joins a watchdog cluster anytime, I
>>> > think this is not a behavior we expect.
>>> >
>>> > Can we make FAILOVER_COMMAND_FINISH_TIMEOUT longer or disable the
>>> > expiring when failover_require_consensus is on?
>>>
>>> Attached is the patch for this.
>>>
>>> > disable the
>>> > expiring when failover_require_consensus is on?
>>>
>>> It seems the patch solves the issue and passed all of regression
>>> test. But I wonder if the patch will give unwanted side effects. What
>>> do you think?
>>>
>>> Best reagards,
>>> --
>>> Tatsuo Ishii
>>> SRA OSS LLC
>>> English: http://www.sraoss.co.jp/index_en/
>>> Japanese:http://www.sraoss.co.jp
>>>
> _______________________________________________
> pgpool-hackers mailing list
> pgpool-hackers at pgpool.net
> http://www.pgpool.net/mailman/listinfo/pgpool-hackers