[pgpool-general: 9088] kind does not match between main(0) slot[0] (52)

[Emond Papegaaij]

Hi,

We've noticed a failure in one of our test runs tonight which I can't
explain. During a reboot test of the nodes in the cluster, one of the
pgpool instances (the one with ip 172.29.30.2) starts returning the
following error:
pid 194: ERROR:  unable to read message kind
pid 194: DETAIL:  kind does not match between main(0) slot[0] (52)

Any attempt to establish a connection to this node fails with the error
above. The other 2 nodes in the cluster are fine. I've attached the log of
the node and copied the status for the watchdog and backends below. What
could be causing this error?

Best regards,
Emond

Watchdog Cluster Information
Total Nodes              : 3
Remote Nodes             : 2
Member Remote Nodes      : 2
Alive Remote Nodes       : 2
Nodes required for quorum: 2
Quorum state             : QUORUM EXIST
Local node escalation    : NO
Leader Node Name         : 172.29.30.2:5432 Linux 8dd701b3cff8
Leader Host Name         : 172.29.30.2

Watchdog Node Information
Node Name         : 172.29.30.1:5432 Linux bd65977729e4
Host Name         : 172.29.30.1
Delegate IP       : Not_Set
Pgpool port       : 5432
Watchdog port     : 9009
Node priority     : 1
Status            : 7
Status Name       : STANDBY
Membership Status : MEMBER

Node Name         : 172.29.30.2:5432 Linux 8dd701b3cff8
Host Name         : 172.29.30.2
Delegate IP       : Not_Set
Pgpool port       : 5432
Watchdog port     : 9009
Node priority     : 1
Status            : 4
Status Name       : LEADER
Membership Status : MEMBER

Node Name         : 172.29.30.3:5432 Linux 7d75ba5595ed
Host Name         : 172.29.30.3
Delegate IP       : Not_Set
Pgpool port       : 5432
Watchdog port     : 9009
Node priority     : 1
Status            : 7
Status Name       : STANDBY
Membership Status : MEMBER

Hostname               : 172.29.30.1
Port                   : 5432
Status                 : 2
Weight                 : 0.333333
Status Name            : up
Backend Status Name    : up
Role                   : standby
Backend Role           : standby
Replication Delay      : 0
Replication State      : streaming
Replication Sync State : async
Last Status Change     : 2024-04-29 04:29:48


Hostname               : 172.29.30.2
Port                   : 5432
Status                 : 2
Weight                 : 0.333333
Status Name            : up
Backend Status Name    : up
Role                   : primary
Backend Role           : primary
Replication Delay      : 0
Replication State      : none
Replication Sync State : none
Last Status Change     : 2024-04-29 04:29:48


Hostname               : 172.29.30.3
Port                   : 5432
Status                 : 2
Weight                 : 0.333333
Status Name            : up
Backend Status Name    : up
Role                   : standby
Backend Role           : standby
Replication Delay      : 0
Replication State      : streaming
Replication Sync State : async
Last Status Change     : 2024-04-29 04:29:48
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20240429/6665ff48/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: container.log
Type: text/x-log
Size: 285863 bytes
Desc: not available
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20240429/6665ff48/attachment.bin>