[pgpool-hackers: 2687] Re: [pgpool-general: 5885] Re: Pgpool-3.7.1 segmentation fault

[Tatsuo Ishii]

Hi Pgpool-II developers,

It is reported that Pgpool-II child process could have a segfault error.

#0  0x000000000042478d in select_load_balancing_node () at
protocol/child.c:1680
1680                    char *database = MASTER_CONNECTION(ses->backend)->sp->database;
(gdb) backtrace
#0  0x000000000042478d in select_load_balancing_node () at
protocol/child.c:1680

To reproduce the problem following conditions should be all met:

1) Streaming replication mode.

2) fail_over_on_backend_error is off.

3) ALWAYS_MASTER flags is set to the master (writer) node.

4) pgpool_status file indicates that the node mentioned in #2 is in
   down status.

What happens here is,

1) find_primary_node() returns node id 0 without checking the status
   of node 0 since ALWAYS_MASTER is set. It's remembered as the
   primary node id. The node id is stored in Req_info->primary_node_id.

2) The connection to backend 0 is not created since pgpool_status says
   it's in down status.

3) upon starting of session, select_load_balancing_node () is called
   and it tries to determine the database name from client's start up
   packet.

4) Since MASTER_CONNECTION macro points to the PRIMARY_NODE,
   MASTER_CONNECTION(ses->backend) is NULL and it results in a segfault.

The fix I propose is, to change PRIMARY_NODE_ID macro so that it
returns REAL_MASTER_NODE_ID (that is the youngest node id which is
alive) if the node id in Req_info->primary_node_id is in down status.
So we have the "true" primary node id in Req_info->primary_node_id,
and "fake" primary node id returned by PRIMARY_NODE_ID macro.

I am afraid it's confusing and may have potential bad effect to
somewhere in Pgpool-II. Note, however, we already let PRIMARY_NODE_ID
return REAL_MASTER_NODE_ID if find_primary_node() cannot find a
primary node. So maybe I am too worried... but I don't know.

So I would like hear opinions from Pgpool-II developers.

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

From: Tatsuo Ishii <ishii at sraoss.co.jp>
Subject: [pgpool-general: 5885] Re: Pgpool-3.7.1 segmentation fault
Date: Wed, 24 Jan 2018 12:00:40 +0900 (JST)
Message-ID: <20180124.120040.507189908198617602.t-ishii at sraoss.co.jp>

>> Thanks for the quick reply! I realized that I ended up in this state,
>> because I was using indexed health checks, and the primary's health checks
>> had been disabled. I've gone back to a single health_check config, to avoid
>> this issue.
> 
> Do you have an issue with "indexed health checks"? I thought it was
> fixed in 3.7.1.
> 
> I've also added an extra pre-start step, which removes the
>> pgpool_status file.
> 
> That might be a solution but I would like to add a guard to Pgpool-II
> against the segfault. The segfault occurs when conditions below are
> all met:
> 
> 1) fail_over_on_backend_error is off.
> 2) ALWAYS_MASTER flags is set to the master (writer) node.
> 
> Attached patch implements the guard against the segfault. Developers
> will start a discussion regarding the patch in pgpool-hackers.
> 
> Best regards,
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
> 
>> On Tue, Jan 23, 2018 at 5:49 PM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>> 
>>> Hi Philip,
>>>
>>> > Hello poolers,
>>> >
>>> > I've compiled pgpool-3.7.1 (./configure --with-openssl; libpq.5.9), for
>>> > Ubuntu 14.04, to connect to RDS Aurora Postgres (9.6.3). When I try to
>>> > authenticate, pgpool child process segfaults. My config file follows the
>>> > instructions set forth by the aurora instructions
>>> > <http://www.pgpool.net/docs/latest/en/html/example-aurora.html>, I
>>> think?
>>> > Have I misconfigured something, to cause this segfault?
>>> >
>>> > Any guidance would be appreciated!
>>> >
>>> > Thanks,
>>> > Philip
>>> >
>>> >
>>> > $ psql -h localhost -U user staging
>>> > Password for user user:
>>> > psql: server closed the connection unexpectedly
>>> >         This probably means the server terminated abnormally
>>> >         before or while processing the request.
>>>
>>> It seems your status file (/var/log/pgpool/pool_status) is out of
>>> date.
>>>
>>> > 2018-01-23 19:23:42: pid 19872: DEBUG:  creating new connection to
>>> backend
>>> > 2018-01-23 19:23:42: pid 19872: DETAIL:  skipping backend slot 0 because
>>> > backend_status = 3
>>>
>>> So Pgpool-II failes to create a connection to backend0, which causes
>>> the segfault later on. Surely Pgpool-II needs to have a guard for the
>>> situation, but for now you could workaround this by shutting down
>>> pgpool, removing /var/log/pgpool/pool_status, and restarting pgoool.
>>>
>>> Once proper pool_status is created, you don't need to repeat the
>>> operation above. i.e. skip removing pool_status.
>>>
>>> Best regards,
>>> --
>>> Tatsuo Ishii
>>> SRA OSS, Inc. Japan
>>> English: http://www.sraoss.co.jp/index_en.php
>>> Japanese:http://www.sraoss.co.jp
>>>