[pgpool-committers: 3948] pgpool: Enhancing the handling of split-brain scenario by the watchdog.

[Muhammad Usama]

Enhancing the handling of split-brain scenario by the watchdog.

Previously, the watchdog cluster was used to call for re-election of the
master/coordinator node whenever the split-brain situation was detected. And
consequently every node was required to rejoin the watchdog network, Which was
essentially similar to the re-booting of the whole watchdog cluster.

Although the technique of calling for re-election at split-brain is good enough
to tackle the situation, but it is very inefficient and prone to the service
disruption for the time when the cluster is rebooting. Also in some cases, like
when the split-brain happens because of temporary network partitioning the new
elections could elect master that has the outdated or invalid backend states.

Now with this enhancement when the cluster detects the split-brain, instead of
blindly calling for new master node election and rebooting every node,
The watchdog nodes try to solve the master/coordinator node contention by
choosing the best candidate.

The candidate for the master/coordinator node is selected on the following criteria.

1-- When two watchdog nodes are claiming to be the cluster master, the master
node that has performed the escalation keeps the master status and the other
node is asked to step down.

2-- If the conflict could not be resolved by the escalation status of the nodes,
The node which holds the quorum remains the master/coordinator.

3-- If the quorum status of both contenders is also same. The node with higher
number of connected alive nodes get the preference.

4-- Finally, if all above three yields no winner, the older master (The node
that has the coordinator status for longer duration) remains the master.

Branch
------
V3_5_STABLE

Details
-------
http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=ab32ae7598ca6476d890882c97e97364b8a7dc6c

Modified Files
--------------
src/include/watchdog/watchdog.h     |   8 +-
src/include/watchdog/wd_json_data.h |   8 +
src/watchdog/watchdog.c             | 512 +++++++++++++++++++++++++++++++++---
src/watchdog/wd_json_data.c         | 117 +++++++-
4 files changed, 609 insertions(+), 36 deletions(-)