[pgpool-general: 9399] Re: Pgpool - Failing over from Primary to Secondary for unknown reason.

Sat Mar 15 00:04:38 JST 2025

Hi,

Thanks for looking into this.
After the graceful shutdown/start of all the 3 instances we didn't see any failovers so far.

We are also working with AWS on health check of the servers, they said there was no issue
with all the 3 servers from the health check perspective.

Here is the information from the log files during failovers.  Nothing different compare to the one I posted earlier.

2025-03-12 02:26:12.972: child pid 2602206: DETAIL:  connecting host=[local]
2025-03-12 02:30:01.556: child pid 2602289: LOG:  new connection received
2025-03-12 02:30:01.556: child pid 2602289: DETAIL:  connecting host=[local]
2025-03-12 02:30:01.620: child pid 2602277: LOG:  new connection received
2025-03-12 02:30:01.620: child pid 2602277: DETAIL:  connecting host=[local]
2025-03-12 02:30:02.799: main pid 2602100: LOG:  reaper handler
2025-03-12 02:30:02.800: main pid 2602100: LOG:  reaper handler: exiting normally
2025-03-12 02:30:02.806: main pid 2602100: LOG:  reaper handler
2025-03-12 02:30:02.807: main pid 2602100: LOG:  reaper handler: exiting normally
2025-03-12 02:30:23.610: ITN.TaskScheduler pid 2602301: LOG:  received degenerate backend request for node_id: 0 from pid [2602301]
2025-03-12 02:30:23.610: ITN.TaskScheduler pid 2602301: LOG:  signal_user1_to_parent_with_reason(0)
2025-03-12 02:30:23.610: main pid 2602100: LOG:  Pgpool-II parent process received SIGUSR1
2025-03-12 02:30:23.610: main pid 2602100: LOG:  Pgpool-II parent process has received failover request
2025-03-12 02:30:23.610: main pid 2602100: LOG:  === Starting degeneration. shutdown host 0.00.00.00(5432) ===
2025-03-12 02:30:23.616: main pid 2602100: LOG:  Restart all children
2025-03-12 02:30:23.619: main pid 2602100: LOG:  execute command: FAIOVER_SCRIPT.sh

2025-03-12 05:05:03.346: main pid 2603109: LOG:  reaper handler
2025-03-12 05:05:03.347: main pid 2603109: LOG:  reaper handler: exiting normally
2025-03-12 05:05:03.358: main pid 2603109: LOG:  reaper handler
2025-03-12 05:05:03.358: main pid 2603109: LOG:  reaper handler: exiting normally
2025-03-12 05:06:20.922: ITN.TaskScheduler pid 2603744: LOG:  received degenerate backend request for node_id: 0 from pid [2603744]
2025-03-12 05:06:20.922: ITN.TaskScheduler pid 2603744: LOG:  signal_user1_to_parent_with_reason(0)
2025-03-12 05:06:20.922: main pid 2603109: LOG:  Pgpool-II parent process received SIGUSR1
2025-03-12 05:06:20.922: main pid 2603109: LOG:  Pgpool-II parent process has received failover request
2025-03-12 05:06:20.922: main pid 2603109: LOG:  === Starting degeneration. shutdown host 0.0.00.00(5432) ===
2025-03-12 05:06:20.925: main pid 2603109: LOG:  Restart all children
2025-03-12 05:06:20.927: main pid 2603109: LOG:  execute command: FAILOVER_SCRIPT.sh

Thanks,
Susihl...
________________________________
From: Bo Peng <pengbo at sraoss.co.jp>
Sent: Friday, March 14, 2025 4:15 AM
To: Sushil Shirodkar <sushilps at hotmail.com>
Cc: pgpool-general at pgpool.net <pgpool-general at pgpool.net>
Subject: Re: [pgpool-general: 9393] Pgpool - Failing over from Primary to Secondary for unknown reason.

Hi,

Thank you for sharing the logs.

The logs don't provide any cause for the failover.
Could you share more logs from other processes before the failover?"
Do you see any health check errors in the logs?

On Wed, 12 Mar 2025 08:01:15 +0000
Sushil Shirodkar <sushilps at hotmail.com> wrote:

> Hi there,
>
> Here is the info. Related to PID 2600121.
> -------------------------------------------------------------------------------------------------------------------
> 2025-03-12 01:10:27.296: child pid 2600121: LOG:  new connection received
> 2025-03-12 01:10:27.296: child pid 2600121: DETAIL:  connecting host=0.0.00.00 port=51121
>
> 2025-03-12 01:39:34.138: IS.Server pid 2600121: LOG:  received degenerate backend request for node_id: 0 from pid [2600121]
> 2025-03-12 01:39:34.138: IS.Server pid 2600121: LOG:  signal_user1_to_parent_with_reason(0)
>
> 2025-03-12 01:39:34.138: main pid 2598786: LOG:  Pgpool-II parent process received SIGUSR1
> 2025-03-12 01:39:34.138: main pid 2598786: LOG:  Pgpool-II parent process has received failover request
> 2025-03-12 01:39:34.138: main pid 2598786: LOG:  === Starting degeneration. shutdown host 0.0.00.00(5432) ===
> 2025-03-12 01:39:34.141: main pid 2598786: LOG:  Restart all children
> 2025-03-12 01:39:34.142: main pid 2598786: LOG:  execute command: FAILOVER_SCRIPT.sh
> --------------------------------------------------------------------------------------------------------------------------
>
> It happened again basically 4th time.  This time  I have shutdown/started all the 3 instances. Will see if that helps.
>
>
> Thanks,
> Sushil...
> ________________________________
> From: Bo Peng <pengbo at sraoss.co.jp>
> Sent: Wednesday, March 12, 2025 2:05 AM
> To: Sushil Shirodkar <sushilps at hotmail.com>
> Cc: pgpool-general at pgpool.net <pgpool-general at pgpool.net>
> Subject: Re: [pgpool-general: 9393] Pgpool - Failing over from Primary to Secondary for unknown reason.
>
> Hi,
>
> Thank you for sharing the logs.
>
> The information related to failover was not output in these logs.
> Could you share more logs related to the process with PID 2600121?
>
> On Wed, 12 Mar 2025 03:57:56 +0000
> Sushil Shirodkar <sushilps at hotmail.com> wrote:
>
> > The following appeared before the failover, nothing abnormal.  Similar lines in previous 1 hour of log file.
> >
> > 2025-03-12 01:35:01.745: child pid 2599666: LOG:  new connection received
> > 2025-03-12 01:35:01.745: child pid 2599666: DETAIL:  connecting host=[local]
> > 2025-03-12 01:35:01.807: child pid 2600001: LOG:  new connection received
> > 2025-03-12 01:35:01.808: child pid 2600001: DETAIL:  connecting host=[local]
> > 2025-03-12 01:35:03.000: main pid 2598786: LOG:  reaper handler
> > 2025-03-12 01:35:03.001: main pid 2598786: LOG:  reaper handler: exiting normally
> > ________________________________
> > From: Bo Peng <pengbo at sraoss.co.jp>
> > Sent: Tuesday, March 11, 2025 11:42 PM
> > To: Sushil Shirodkar <sushilps at hotmail.com>
> > Cc: pgpool-general at pgpool.net <pgpool-general at pgpool.net>
> > Subject: Re: [pgpool-general: 9393] Pgpool - Failing over from Primary to Secondary for unknown reason.
> >
> > Hi,
> >
> > To figure out the cause of the failover, please check the
> > Pgpool-II logs right before the failover occurred.
> > Did any error occur?
> >
> > On Wed, 12 Mar 2025 03:22:45 +0000
> > Sushil Shirodkar <sushilps at hotmail.com> wrote:
> >
> > >
> > > Hello all,
> > >
> > > We have 3 servers env. Pgpool + Master/Slave on AWS,  for some reason Pgpool  thinks Master is down and promotes Slave as Read/Write.   Log file on Pgpool shows the below mentioned message during the failover.  We had failure thrice in a single day so far today on this production env.  with diff.  pids, nothing changed in last couple of months.
> > >
> > > TS_SERVER pid 2600121 LOG-received degenerate backend request for node_id- 0 from pid 2600121
> > >
> > > Env:
> > > Pgpool-II => 4.5.2
> > > Postgres => 16.4
> > > Ubuntu   => 22.04.4
> > >
> > > Would be helpful if you folks share your thoughts on the above.
> > >
> > >
> > > Thanks,
> > > Sushil....
> >
> >
> > --
> > Bo Peng <pengbo at sraoss.co.jp>
> > SRA OSS K.K.
> > TEL: 03-5979-2701 FAX: 03-5979-2702
> > URL: https://www.sraoss.co.jp/
>
>
> --
> Bo Peng <pengbo at sraoss.co.jp>
> SRA OSS K.K.
> TEL: 03-5979-2701 FAX: 03-5979-2702
> URL: https://www.sraoss.co.jp/

--
Bo Peng <pengbo at sraoss.co.jp>
SRA OSS K.K.
TEL: 03-5979-2701 FAX: 03-5979-2702
URL: https://www.sraoss.co.jp/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20250314/8f56a40d/attachment.htm>