[pgpool-hackers: 3434] [Proposal] more fault injection testing for watchdog

[Tatsuo Ishii]

Currently we have a fault injection mechanism to test communication
failure between health check process and PostgreSQL backend. This is
handy to test the quarantine state handling in watchdog.

Recently there are multiple reports from field that temporary heartbeat
dead errors cause unexpected behavior of split-brain resolution[1][2].
Hence I would like to propose to test those faults by inventing a new
fault injection mechanism for watchdog to simulate heartbeat dead
errors. One of the ideas is, to create a file under log directory. And
if the life-check process finds the file, it does not update the state
"received heartbeat", which eventually causes heartbeat dead error.

With this feature, we could test watchdog heartbeat dead scenario by
using watchdog_setup on a local computer, which is very handy for
developers, and thus it makes possible to add such that test case to
the regression test.

[1] https://www.pgpool.net/mantisbt/view.php?id=545
[2] https://www.pgpool.net/mantisbt/view.php?id=547

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp