5.14. Watchdog

If on, activates the watchdog. Default is off

This parameter can only be set at server start.

Specifies the hostname or IP address of Pgpool-II server. This is used for sending/receiving queries and packets, and also as an identifier of the watchdog node.

This parameter can only be set at server start.

Specifies the port number to be used by watchdog process to listen for connections. Default is 9000.

This parameter can only be set at server start.

Specifies the authentication key used for all watchdog communications. All Pgpool-II must have the same key. Packets from watchdog having different key will get rejected. This authentication is also applied to the heartbeat signals when the heartbeat mode is used as a lifecheck method.

Since in Pgpool-IIV3.5 or beyond wd_authkey is also used to authenticate the watchdog IPC clients, all clients communicating with Pgpool-II watchdog process needs to provide this wd_authkey value for "IPCAuthKey" key in the JSON data of the command.

Default is '' (empty) which means disables the watchdog authentication.

This parameter can only be set at server start.

Specifies the list of trusted servers to check the up stream connections. Each server in the list is required to respond to ping. Specify a comma separated list of servers such as "hostA,hostB,hostC". If none of the server are reachable, watchdog will regard it as failure of the Pgpool-II. Therefore, it is recommended to specify multiple servers. Please note that you should not assign PostgreSQL servers to this parameter.

This parameter can only be set at server start.

Specifies the path of a ping command for monitoring connection to the upper servers. Set the only path of the directory containing the ping utility, such as "/bin" or such directory.

This parameter can only be set at server start.

Specifies the virtual IP address (VIP) of Pgpool-II that is connected from client servers (application servers etc.). When a Pgpool-II is switched from standby to active, the Pgpool-II takes over this VIP. VIP will not be brought up in case the quorum does not exist. Default is ''(empty): which means virtual IP will never be brought up.

This parameter can only be set at server start.

Specifies the path to the command that Pgpool-II will use to switch the virtual IP on the system. Set only the path of the directory containing the binary, such as "/sbin" or such directory.

This parameter can only be set at server start.

Specifies the command to bring up the virtual IP. Set the command and parameters such as "ip addr add $_IP_$/24 dev eth0 label eth0:0" $_IP_$ will get replaced by the IP address specified in the delegate_IP.

This parameter can only be set at server start.

Specifies the command to bring down the virtual IP. Set the command and parameters such as "ip addr del $_IP_$/24 dev eth0".

This parameter can only be set at server start.

Specifies the path to the command that Pgpool-II will use to send the ARP requests after the virtual IP switch. Set only the path of the directory containing the binary, such as "/usr/sbin" or such directory.

This parameter can only be set at server start.

Specifies the command to use for sending the ARP requests after the virtual IP switch. Set the command and parameters such as "arping -U $_IP_$ -w 1". $_IP_$ will get replaced by the IP address specified in the delegate_IP.

This parameter can only be set at server start.

When set to on, watchdog clears all the query cache in the shared memory when pgpool-II escalates to active. This prevents the new active Pgpool-II from using old query caches inconsistent to the old active.

Default is on.

This works only if memqcache_method is 'shmem'.

This parameter can only be set at server start.

Watchdog executes this command on the node that is escalated to the master watchdog.

This command is executed just before bringing up the virtual IP if that is configured on the node.

This parameter can only be set at server start.

Watchdog executes this command on the master Pgpool-II watchdog node when that node resigns from the master node responsibilities. A master watchdog node can resign from being a master node, when the master node Pgpool-II shuts down, detects a network blackout or detects the lost of quorum.

This command is executed before bringing down the virtual/floating IP address if it is configured on the watchdog node.

wd_de_escalation_command is not available prior to Pgpool-II V3.5.

This parameter can only be set at server start.

When enabled, Pgpool-II will perform the degenerate/failover on backend node if the quorum exists. In the absence of the quorum, Pgpool-II node that detects the backend failure will quarantine the failed backend node until the quorum exists again.

The quarantine nodes behaves similar to the detached backend nodes but unlike failed/degenerated backends the quarantine status is not propagated to the other Pgpool-II nodes in the watchdog cluster, So even if the backend node is in the quarantine state on one Pgpool-II node, other Pgpool-II nodes may still continue to use that backend.

Although there are many similarities in quarantine and failover operations, but they both differ in a very fundamental way. The quarantine operations does not executes the failover_command and silently detaches the problematic node, So in the case when the master backend node is quarantined, the Pgpool-II will not promote the standby to take over the master responsibilities and until the master node is quarantined the Pgpool-II will not have any usable master backend node.

Default is on.

failover_when_quorum_exists is not available prior to Pgpool-II V3.7.

This parameter can only be set at server start.

When enabled, Pgpool-II will perform the degenerate/failover on a backend node if the watchdog quorum exists and at-least minimum number of nodes necessary for the quorum vote for the failover.

For example, in a three node watchdog cluster, the failover will only be performed until at least two nodes ask for performing the failover on the particular backend node.

Default is on.

failover_require_consensus is not available prior to Pgpool-II V3.7. and it is only effective when failover_when_quorum_exists is enabled

This parameter can only be set at server start.

This parameter works in connection with the failover_require_consensus. When enabled, a single Pgpool-II node can cast multiple votes for the failover.

For example, in a three node watchdog cluster, if one Pgpool-II node sends two failover requests for a particular backend node failover, Both requests will be counted as a separate vote in the favor of the failover and Pgpool-II will execute the failover, even if it does not get the vote from any other Pgpool-II node.

Default is off.

allow_multiple_failover_requests_from_node is not available prior to Pgpool-II V3.7. and it is only effective when both failover_when_quorum_exists and failover_require_consensus are enabled

This parameter can only be set at server start.

Specifies the method of life check. This can be either of 'heartbeat' (default), 'query' or 'external'.

heartbeat: In this mode, watchdog sends the heartbeat signals (UDP packets) periodically to other Pgpool-II. Similarly watchdog also receives the signals from other Pgpool-II . If there are no signal for a certain period, watchdog regards is as failure of the Pgpool-II .

query: In this mode, watchdog sends the monitoring queries to other Pgpool-II and checks the response

external: This mode disables the built in lifecheck of Pgpool-II watchdog and relies on external system to provide node health checking of local and remote watchdog nodes.

external mode is not available in versions prior to Pgpool-II V3.5.

This parameter can only be set at server start.

Specify a comma separated list of network device names, to be monitored by the watchdog process for the network link state. If all network interfaces in the list becomes inactive (disabled or cable unplugged), the watchdog will consider it as a complete network failure and the Pgpool-II node will commit the suicide. Specifying an ''(empty) list disables the network interface monitoring. Setting it to 'any' enables the monitoring on all available network interfaces except the loopback. Default is '' empty list (monitoring disabled).

wd_monitoring_interfaces_list is not available in versions prior to Pgpool-II V3.5.

This parameter can only be set at server start.

Specifies the interval between life checks of Pgpool-II in seconds. (A number greater than or equal to 1) Default is 10.

This parameter can only be set at server start.

This parameter can be used to elevate the local watchdog node priority in the elections to select master watchdog node. The node with the higher wd_priority value will get selected as master watchdog node when cluster will be electing its new master node in the event of old master watchdog node failure. wd_priority is also valid at the time of cluster startup. When some watchdog nodes start up at same time,a node with the higher wd_priority value is selected as a master node. So we should start watchdog nodes in order of wd_priority priority to prevent unintended nodes from being selected as master. If wd_priority of each nodes has the same value, leader node will be decided on base of starting time of watchdog.

wd_priority is not available in versions prior to Pgpool-II V3.5.

This parameter can only be set at server start.

The directory where the UNIX domain socket accepting Pgpool-II watchdog IPC connections will be created. Default is '/tmp'. Be aware that this socket might be deleted by a cron job. We recommend to set this value to '/var/run' or such directory.

wd_ipc_socket_dir is not available in versions prior to Pgpool-II V3.5.

This parameter can only be set at server start.

Specifies the UDP port number to receive heartbeat signals. Default is 9694. wd_heartbeat_port is only applicable if the wd_lifecheck_method is set to 'heartbeat'

This parameter can only be set at server start.

Specifies the interval time in seconds between sending the heartbeat signals. Default is 2. wd_heartbeat_keepalive is only applicable if the wd_lifecheck_method is set to 'heartbeat'

This parameter can only be set at server start.

Specifies the time in seconds before marking the remote watchdog node as failed/dead node, if no heartbeat signal is received within that time. Default is 30 wd_heartbeat_deadtime is only applicable if the wd_lifecheck_method is set to 'heartbeat'

This parameter can only be set at server start.

Specifies the IP address or hostname of destination the remote Pgpool-II for sending the heartbeat signals. Multiple destinations can be configured for the heartbeat signals, the number at the end of the parameter name is referred as the "destination number", that starts from 0.

heartbeat_destination is only applicable if the wd_lifecheck_method is set to 'heartbeat'

This parameter can only be set at server start.

Specifies the destination port number of the remote Pgpool-II for sending the heartbeat signals. Multiple destinations can be configured for the heartbeat signals, the number at the end of the parameter name is referred as the "destination number", that starts from 0.

heartbeat_destination_port is only applicable if the wd_lifecheck_method is set to 'heartbeat'

This parameter can only be set at server start.

Specifies the network device name for sending the heartbeat signals to the destination specified by heartbeat_destinationX:heartbeat_destination_portX Different heartbeat devices can be configured for each heartbeat destination by changing the value of X(destination number). at the end of parameter name. The destination index number starts from 0.

heartbeat_device is only applicable if Pgpool-II is started with root privilege. If not, leave it as an empty string ('').

heartbeat_device is only applicable if the wd_lifecheck_method is set to 'heartbeat'

This parameter can only be set at server start.

Specifies the number of times to retry a failed life check of pgpool-II. Valid value could be a number greater than or equal to 1. Default is 3.

wd_life_point is only applicable if the wd_lifecheck_method is set to 'query'

This parameter can only be set at server start.

Specifies the query to use for the life check of remote Pgpool-II. Default is "SELECT 1".

wd_lifecheck_query is only applicable if the wd_lifecheck_method is set to 'query'

This parameter can only be set at server start.

Specifies the database name for the connection used for the life check of remote Pgpool-II. Default is "template1".

wd_lifecheck_dbname is only applicable if the wd_lifecheck_method is set to 'query'

This parameter can only be set at server start.

Specifies the user name for the connection used for the life check of remote Pgpool-II. Default is "nobody".

wd_lifecheck_user is only applicable if the wd_lifecheck_method is set to 'query'

This parameter can only be set at server start.

Specifies the password for the user used for the life check of remote Pgpool-II.

If wd_lifecheck_password is left blank Pgpool-II will first try to get the password for wd_lifecheck_user from pool_passwd file before using the empty password.

You can also specify AES256-CBC encrypted password in wd_lifecheck_password field. To specify the AES encrypted password, password string must be prefixed with AES after encrypting (using aes-256-cbc algorithm) and encoding to base64.

To specify the unencrypted clear text password, prefix the password string with TEXT. For example if you want to set mypass as a password, you should specify TEXTmypass in the password field. In the absence of a valid prefix, Pgpool-II will considered the string as a plain text password.

You can also use pg_enc utility to create the correctly formatted AES encrypted password strings.

wd_lifecheck_password is only applicable if the wd_lifecheck_method is set to 'query'

This parameter can only be set at server start.

Default is ''(empty).

Specifies the hostname of remote Pgpool-II server for watchdog node. The number at the end of the parameter name is referred as "server id", and it starts from 0.

This parameter can only be set at server start.

Specifies the port number of the remote Pgpool-II server for watchdog node. The number at the end of the parameter name is referred as "server id", and it starts from 0.

This parameter can only be set at server start.

Specifies the watchdog port number of the remote Pgpool-II server for watchdog node. The number at the end of the parameter name is referred as "server id", and it starts from 0.

This parameter can only be set at server start.