[pgpool-committers: 7917] pgpool: Implementing the follow_primary command-locking over the watchd

[Muhammad Usama]

Implementing the follow_primary command-locking over the watchdog channel.

Supplementary fix for [pgpool-hackers: 3892] Problem with detach_false_primary..

commit:455f00dd5f5b7b94bd91aa0b6b40aab21dceabb9 fixed a race condition between
detach_false_primary and follow_primary commands. Part of the fix was to make
sure that the detach_false_primary should only be executed on the
leader watchdog node.

The mentioned commit ensures the execution of detach_false_primary on the
watchdog leader by getting the watchdog status from within the main process.
The design is good enough for most cases, but has the potential to fail if
the cluster goes into the election process just after the main process
has read the status.

To fix that, this commit implements the synchronization of follow_primary_command
execution using the distributed locks over the watchdog channel.

The idea is, just before executing the follow_primary during the failover process
we instruct all standby watchdog nodes to acquire a lock on their respective
nodes to block the false primary detection during the period when the
follow_primary is being executed on the leader watchdog node.

Moreover to keep the watchdog process blocked on waiting for the lock the commit
introduced the pending remote lock mechanism, so that remote locks can get
acquired in the background after the completion of the inflight replication checks.

Finally, REQ_DETAIL_CONFIRMED flag is removed from degenerate_backend_set()
request that gets issued to detach the false primary, That means all quorum
and consensus rules must be satisfied for the detach to happen.

Branch
------
V4_2_STABLE

Details
-------
https://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=f50cf64c5fe70e27afc6bc8fdfb1c0bb6e762262
Author: Muhammad Usama <m.usama at highgo.ca>

Modified Files
--------------
src/include/pool.h                            |  16 +++-
src/include/watchdog/wd_internal_commands.h   |  17 ++++
src/include/watchdog/wd_ipc_defines.h         |   1 +
src/main/main.c                               |   1 +
src/main/pgpool_main.c                        | 117 ++++++++++++++++++++++++--
src/streaming_replication/pool_worker_child.c |  18 ++--
src/watchdog/watchdog.c                       |  78 ++++++++++++++++-
src/watchdog/wd_internal_commands.c           |  36 ++++++++
8 files changed, 261 insertions(+), 23 deletions(-)