pgpool-II 3.5 watchdog test

Test Num	Category	Test Description	Expected Output	How to test	Assigned-to
1	Installation	Make sure that the new watchdog is installed and configured successfully
2	Upgrade	Make sure that pgpool II with new watchdog can be installed on a system running pgpool II with the old watchdog
3	Configuration	Make sure that pgpool II can be configured successfully with one primary and one stand-by configuration
4	Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a sample query	Configruation
4.1	Functional testing	Shutdown Host-1's pgpool instanse and execute query again	Host-2 should take and respond to query
4.2	Functional testing	Shutdown Host-2's pgpool instanse and execute query again	Host-3 should take and respond to query
4.3	Functional testing	Start Host-1's pgpool instanse and execute query again	Need to see which host will respond to query
5	Failover scenarions / Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a sample query	Configruation
5.1	Failover scenarios	Un-Plug Host-1's network cable and execute query again	Host-2 should take and respond to query
5.2	Failover scenarios	Un-Plug Host-2's network cable and execute query again	Host-3 should take and respond to query
5.3	Failover scenarios	Plug Host-1's network cable execute query again	Need to see which host will respond to query
6	Functional testing / Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a long query
6.1	Functional testing	Shutdown / Power Off Host-1's instanse and execute query again	Host-2 should take over and start responding, need to see the already running query response.
6.2	Functional testing	Shutdown / Power Off Host-2's instanse and execute query again	Host-3 should take over and start responding, need to see the already running query response.
6.3	Functional testing	Start Host-1's pgpool instanse and execute query again	Need to see which host will respond to query
7.1	Cheking other functionality of watchdog	Changing active/standby state in case of certain faults detected
7.2	Cheking other functionality of watchdog	Automatic virtual IP address assigning synchronous to server switching
7.3	Cheking other functionality of watchdog	Automatic registration of a server as standby in recovery
8	Isolated master scenario / Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a query
8.1	Isolated master scenario	Break the connectivity between pgpool watchdog primary and stand node by bringing down connectivity on the stand-by.	Split brain testing ensures that their is only one master at a time that the clients can connect to. The stand-by should be promoted as the primary and the clients shouldn't be able to connect to the old master.
9	Networking Isolation scenarion
9.1
10	Database Failure
10.1
10.2
11	Watchdog agent failure
11.1
11.2
12	Pgpool-II Watchdog integration
12.1	PG backend failover with watchdog	Test the command interlocking, when pgpool-II watchdog in enabled.	The failover and follow master scripts should only be executed by one pgpool-II node
12.2	PG backend failback with watchdog	Test the command interlocking, when pgpool-II watchdog in enabled.	The failback scripts should only be executed by one pgpool-II node
12.3	online recovery with watchdog enabled	Execute online recovery with watchdog enabled.
12.4	pgpool-II configuration integrity	Perform test with changing pgpool-II configurations on different pgpool-II nodes	Standby watchdog shoud report and fail to start if the configurations on master node is differnet from this node
13	integrate external node health checking	Test if watchdog can successfully integrate with external health-checking system	send the node down and node alive messages to watchdog ipc socket and it should handle these appropriately