[pgpool-general: 8634] PgPool-II throws 'kind mismatch among backends'

[leis_byways_00 at icloud.com]

Greetings everyone,

I’m running a 3 replica PostgreSQL cluster with Repmgr and each a PgPool-II in front in streaming replication mode.

This database is used by a GitLab deployment which comes with its own backup tool that snapshots via pg_dump and irregularly when this operation is executed I’m getting the following error:

2023-03-05 00:00:38 UTC -- Dumping database ...
pg_dump: error: query failed: FATAL:  failed to read kind from backend
DETAIL:  kind mismatch among backends. Possible last query was: "SET TRANSACTION SNAPSHOT '00000059-0001C517-1'" kind details are: 0[E: invalid snapshot identifier: "00000059-0001C517-1"] 1[C]
HINT:  check data consistency among db nodes
server closed the connection unexpectedly
	This probably means the server terminated abnormally
	before or while processing the request.
pg_dump: detail: Query was: SET TRANSACTION SNAPSHOT '00000059-0001C517-1'
Dumping PostgreSQL database gitlabhq_production ... 2023-03-05 00:00:38 UTC -- Dumping database failed: Failed to create compressed file '/srv/gitlab/tmp/backups/db/database.sql.gz' when trying to backup the main database:
 - host: 'pg-pgpool.gitlab-postgres.svc'
 - port: '5432'
 - database: ‘gitlabhq_production’

My initial thought was that something is not right with the replication, so I’ve executed on PgPool-II the following command to check for replication issues:
psql (14.5)
Type "help" for help.

postgres=# show pool_nodes;
 node_id |                hostname                | port | status | pg_status | lb_weight |  role   | pg_role | select_cnt | load_balance_node | replication_delay | replication_state | replication_sync_state | last_status_change
---------+----------------------------------------+------+--------+-----------+-----------+---------+---------+------------+-------------------+-------------------+-------------------+------------------------+---------------------
 0       | pg-postgresql-0.pg-postgresql-headless | 5432 | up     | up        | 0.333333  | primary | primary | 93         | false             | 0                 |                   |                        | 2023-02-14 13:02:35
 1       | pg-postgresql-1.pg-postgresql-headless | 5432 | up     | up        | 0.333333  | standby | standby | 62         | false             | 0                 |                   |                        | 2023-02-14 13:03:15
 2       | pg-postgresql-2.pg-postgresql-headless | 5432 | up     | up        | 0.333333  | standby | standby | 86         | true              | 0                 |                   |                        | 2023-02-14 13:03:15
(3 rows)

Further searches on the Internet took me to the point that, in streaming replication mode, the columns replication_state and replication_sync_state have to be populated, as per this support document.
Now my confusion is, what I’m facing currently, is this an aftermath of the above mentioned fields not being populated? I’m not really familiar with how replication is tackled on PgPool-II, does anyone have any thoughts as what could cause such issues?
Best regards,
~Daniel M.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20230311/a063b6c7/attachment.htm>