[pgpool-committers: 6304] pgpool: Fix for miscellaneous watchdog issues.

[Muhammad Usama]

Fix for miscellaneous watchdog issues.

The commit takes care of following reports in watchdog
-- pgpool-general: 6672 Query (watchdog split brain)
-- 0000547: We need to do arping again after recovering from split-brain.

Basically in a bid to solve these issue, the commit makes blow mentioned
behavioral changes in watchdog

1-- If life check reports a watchdog node is dead but the watchdog core is still
able to connect and communicate with it without any problem, the watchdog core
was getting into the dilemma of whether to consider it as a lost or an alive node.

Fix:
With this commit, the lost nodes reported by life-check (external or internal)
are treated as hard failures even when the node is reachable from the watchdog
core. and that node can only become alive if it is restarted or secondly if
life-check informs that the node has become alive again.
The second type of node failures that can mark the watchdog node as lost, occurs
when a watchdog node fails to respond to the messages sent to it or frequent
errors occur on its socket. These errors are detected by the watchdog core
itself and get recovered automatically whenever that node becomes responsive
again. Apart from this the commit also made some improvements in the area of
detecting such internal errors.

2-- Standby nodes were too aggressive in reacting to the coup attempt
(when another watchdog node tries to become a master while valid master already
exists in the cluster) or to the possibility of split-brain (when it receives
"I AM MASTER" message from the node that is not the master as per the standby
node record). In both these situations, the standby nodes used to re-join
the master node in the hope to find the true master.
But that didn't prove to be a very good strategy, since it may lead to
influence the selection of true master node when the true-master and
fake-master nodes get into a fight to retain the master node status.
As one of the things true and fake master compares is the number of connected
standby nodes. And if during the fight the standby leaves the current master
to re-join the cluster the standby node count on the true master becomes
lower than actual and that affects its chances of winning the elections.

Fix:
The commit makes the standby nodes more lean-back and to actively reject the
nodes that are trying to become a master in the presence of the true-master
and avoid re-joining the cluster until it is absolutely necessary.

3-- The third problem was in the case of network partitioning or partial
life-check failure (when node A thinks node B is lost but node B thinks
node A is not lost) In this case the kicked out standby node was too
aggressive in trying to connect to the master or become a master itself
so that potentially put the unnecessary burden on the network and the
cluster nodes.

Fix:
So the fix for this issue is to make the isolated node a bit calm and wait
between trying to connect to master or become a master. For that purpose,a new
WD_NETWORK_ISOLATION state is added to the state machine, which just adds a
delay between successive tries of becoming a master. The node can only go into
this state when it comes to knowing that it is marked as lost on the current
master and gets out of it when it gets an indication that life-check on the
remote node has informed that node is alive again or secondly it gets out of
it after 10 seconds anyway.

Finally, Because creating the partial or complete network failures scenario is
very difficult so the commit also adds a watchdog debug aid similar to the
health-check debug to simulate different scenarios.

To enable the watchdog-debug aid the Pgpool-II needs to be compiled with
WATCHDOG_DEBUG=1" flag ($make WATCHDOG_DEBUG=1)

Once compiled with debug aid enabled, you can put the command in
pgpool_logdir->watchdog_debug_requests file to test different scenarios.

e.g.

$ echo "KILL_ALL_COMMUNICATION" > logdir/watchdog_debug_requests
$ echo "KILL_ALL_SENDERS" >> logdir/watchdog_debug_requests

The current list of commands that are supported by watchdog debug aid are

DO_NOT_REPLY_TO_BEACON <= Standby node stops replying master node beacon messages
            while this line is present in the watchdog_debug_requests file
DO_NOT_SEND_BEACON <= Master node stops sending beacon messages to standby nodes
            while this line is present in the watchdog_debug_requests file
KILL_ALL_COMMUNICATION <= Watchdog stops all communication with all nodes
            while this line is present in the watchdog_debug_requests file
KILL_ALL_RECEIVERS  <= Watchdog ignores messages of all node
            while this line is present in the watchdog_debug_requests file
KILL_ALL_SENDERS  <= Watchdog stops sending messages to all nodes
            while this line is present in the watchdog_debug_requests file

Branch
------
V3_7_STABLE

Details
-------
https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=88d138f06927c13e2516262ef17395bedd6c1faa

Modified Files
--------------
src/include/watchdog/watchdog.h |  31 +-
src/test/regression/regress.sh  |   2 +-
src/watchdog/Makefile.am        |   3 +-
src/watchdog/Makefile.in        |   3 +-
src/watchdog/watchdog.c         | 755 ++++++++++++++++++++++++++++++++++------
5 files changed, 690 insertions(+), 104 deletions(-)