What is Watchdog

"watchdog" is a sub process of pgpool-II aiming for adding high availability feature to it. Features added by watchdog include:

Life checking of pgpool service

watchdog monitors responses of pgpool service rather than process. It sends queries to PostgreSQL via pgpool which is being monitored by watchdog and watchdog checks the response. Also watchdog monitors connections to up stream servers (application servers etc.) from the pgpool. The connection between the up stream servers and the pgpool is monitored as service of pgpool.

Mutual monitoring of watchdog processes

watchdog processes exchange information on the monitored servers to keep the information up to date, and to allow watchdog processes to mutually monitor each other.

Changing active/standby state in case of certain faults detected

When a fault is detected in the pgpool service, watchdog notifies the other watchdogs of it. Watchdogs decide the new active pgpool if previous active pgpool is broken by voting and change active/standby state.

Automatic virtual IP address assigning synchronous to server switching

When a standby pgpool server promotes to active, the new active server brings up virtual IP interface. Meanwhile, the previous active server brings down the virtual IP interface. This enables the active pgpool to work using the same IP address even when servers is switched over.

Automatic registration of a server as standby in recovery

When broken server recovers or new server is attached, the watchdog process notifies the other watchdog process along with information of the new server, and the watchdog process receives information on the active server and other servers. Then, the attached server is registered as standby.

Server composition

Figure below describes how pgpool-II and watchdog process is configured.

Starting/stopping watchdog

watchdog process is started/stopped automatically by pgpool-II, therefore there is no dedicated command to start/stop it. However, pgpool-II must to be started with root privilege because watchdog process needs to control the virtual IP interface.

Configuring watchdog (pgpool.conf)

watchdog configuration parameters is described in pgpool.conf. There is sample configuration in WATCHDOG section in pgpool.conf.sample file. All following options are required to be specified in watchdog process.

Basic configuration

use_watchdog: If on, activates watchdog. Default is off.
trusted_servers: The list of trusted servers to check the up stream connections. Each server is required to respond to ping. Specify a comma separated list of servers such as "hostA,hostB,hostC".
delegate_IP: Specifies the virtual IP address (VIP) of pgpool-II that is connected from client servers (application servers etc.). When a pgpool is switched from standby to active, the pgpool takes over this VIP.
wd_hostname: Specifies the hostname or IP address for mutual monitoring of watchdog processes.
wd_port: Specifies the port number for mutual monitoring of watchdog processes.
wd_interval: This parameter specifies the interval between life checks of pgpool-II in second. (A number greater than or equal to 1)
ping_path: This parameter specifies a path of ping command for monitoring connection to the upper servers. Set the only path such as "/bin".
ifconfig_path: This parameter specifies a path of a command to switch the IP address. Set the only path such as "/sbin".
if_up_cmd: This parameter specifies a command to bring up the virtual IP. Set the command and parameters such as "ifconfig eth0:0 inet $_IP_$ netmask 255.255.255.0". $_IP_$ is replaced by the IP address specified in delegate_IP.
if_down_cmd: This parameter specifies a command to bring down the virtual IP. Set the command and parameters such as "ifconfig eth0:0 down".
arping_path: This parameter specifies a path of a command to send an ARP request after the virtual IP is switched. Set the only path such as "/usr/sbin".
arping_cmd: This parameter specifies a command to send an ARP request after the virtual IP is switched. Set the command and parameters such as "arping -U $_IP_$ -w 1". $_IP_$ is replaced by the IP address specified in delegate_IP.
wd_life_point: The times to retry a failed life check of pgpool-II. (A number greater than or equal to 1)
wd_lifecheck_query: Actual query to check pgpool-II. Default is "SELECT 1"

Servers to be monitored

other_pgpool_hostname0: Specifies the hostname pgpool-II server to be monitored. The number at the end of the parameter name is referred as "server id", and it starts from 0.
other_pgpool_port0: Specifies the port number for pgpool on pgpool-II server to be monitored. The number at the end of the parameter name is referred as "server id", and it starts from 0.
other_wd_port0: Specifies the port number for watchdog on pgpool-II server to be monitored. The number at the end of the parameter name is referred as "server id", and it starts from 0.