pgpool-II 3.5 watchdog test

Test Num	Category	Test Description	Expected Output	How to test	Assigned-to
1	Installation	Make sure that the new watchdog is installed and configured successfully
2	Upgrade	Make sure that pgpool II with new watchdog can be installed on a system running pgpool II with the old watchdog
3	Configuration	Make sure that pgpool II can be configured successfully with one primary and one stand-by configuration
4	Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a sample query	Configruation
4.1	Functional testing	Shutdown Host-1's pgpool instanse and execute query again	Host-2 should take and respond to query
4.2	Functional testing	Shutdown Host-2's pgpool instanse and execute query again	Host-3 should take and respond to query
4.3	Functional testing	Start Host-1's pgpool instanse and execute query again	Need to see which host will respond to query
5	Failover scenarions / Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a sample query	Configruation
5.1	Failover scenarios	Un-Plug Host-1's network cable and execute query again	Host-2 should take and respond to query
5.2	Failover scenarios	Un-Plug Host-2's network cable and execute query again	Host-3 should take and respond to query
5.3	Failover scenarios	Plug Host-1's network cable execute query again	Need to see which host will respond to query
6	Functional testing / Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a long query
6.1	Functional testing	Shutdown / Power Off Host-1's instanse and execute query again	Host-2 should take over and start responding, need to see the already running query response.
6.2	Functional testing	Shutdown / Power Off Host-2's instanse and execute query again	Host-3 should take over and start responding, need to see the already running query response.
6.3	Functional testing	Start Host-1's pgpool instanse and execute query again	Need to see which host will respond to query
7.1	Cheking other functionality of watchdog	Changing active/standby state in case of certain faults detected
7.2	Cheking other functionality of watchdog	Automatic virtual IP address assigning synchronous to server switching
7.3	Checking other functionality of watchdog	Automatic registration of a server as standby in recovery
8	Isolated master scenario / Setup	Three pgpool instanses (Host-1, Host-2, Host-3) are running on different machine using Ubutu 13:04. Connect to Host-1 and execute a query
8.1	Isolated master scenario	Break the connectivity between pgpool watchdog primary and stand node by bringing down connectivity on the stand-by.	Split brain testing ensures that their is only one master at a time that the clients can connect to. The stand-by should be promoted as the primary and the clients shouldn't be able to connect to the old master.
9	Networking Isolation scenarion	In this scenario we need to check if the pgpool primary and secondary node loose contact due to interruption in network connectivity. This means that two instances loose connectivity.	The watchdog process should wait based on some timeout value in order to see if the network comes backup. Need to make sure that we don't have a situation where both instances are acting as primary.
10	Testing watchdog on cloud	In this case we want to test watchdog functionality by deploying watchdog, pgpool instances and database server on Amazon cloud. The goal should be perform all the test scenario on AWS that are performed on-premise	All the watchdog functionality should work on AWS similar to how it works in on-premise deployment.
11	Database Failure	In the test case, the PG database running on the secondary dies. This can be done by stopping PG services on the secondary node.	The expected scenario is that pgpool primary node will function as is and continue to serving queries from the client. A pgpool secondary node can be added later on to the watchdog cluster, i believe pgpool needs to be restarted after adding back the secondary node.
12	Database Failure	In the test case, the PG database running on the primary node dies. This can be done by stopping PG services on the primary node.	The expected scenario is that pgpool instance running secondary node will be promoted to primary node and it will serve queries from the client. A secondary node can be added later on, i believe pgpool needs to be restarted after adding back the secondary node.
12	Watchdog agent failure
13
14	Pgpool-II Watchdog integration
15	PG backend failover with watchdog	Test the command interlocking, when pgpool-II watchdog in enabled.	The failover and follow master scripts should only be executed by one pgpool-II node
16	PG backend failback with watchdog	Test the command interlocking, when pgpool-II watchdog in enabled.	The failback scripts should only be executed by one pgpool-II node
17	online recovery with watchdog enabled	Execute online recovery with watchdog enabled.
18	pgpool-II configuration integrity	Perform test with changing pgpool-II configurations on different pgpool-II nodes	Standby watchdog shoud report and fail to start if the configurations on master node is differnet from this node
19	integrate external node health checking	Test if watchdog can successfully integrate with external health-checking system	send the node down and node alive messages to watchdog ipc socket and it should handle these appropriately