<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi, </div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for looking into this. <br>
After the graceful shutdown/start of all the 3 instances we didn't see any failovers so far. </div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
We are also working with AWS on health check of the servers, they said there was no issue </div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
with all the 3 servers from the health check perspective. </div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Here is the information from the log files during failovers. Nothing different compare to the one I posted earlier. </div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:26:12.972: child pid 2602206: DETAIL: connecting host=[local]</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:01.556: child pid 2602289: LOG: new connection received</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:01.556: child pid 2602289: DETAIL: connecting host=[local]</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:01.620: child pid 2602277: LOG: new connection received</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:01.620: child pid 2602277: DETAIL: connecting host=[local]</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:02.799: main pid 2602100: LOG: reaper handler</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:02.800: main pid 2602100: LOG: reaper handler: exiting normally</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:02.806: main pid 2602100: LOG: reaper handler</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:02.807: main pid 2602100: LOG: reaper handler: exiting normally</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:23.610: ITN.TaskScheduler pid 2602301: LOG: received degenerate backend request for node_id: 0 from pid [2602301]</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:23.610: ITN.TaskScheduler pid 2602301: LOG: signal_user1_to_parent_with_reason(0)</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:23.610: main pid 2602100: LOG: Pgpool-II parent process received SIGUSR1</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:23.610: main pid 2602100: LOG: Pgpool-II parent process has received failover request</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:23.610: main pid 2602100: LOG: === Starting degeneration. shutdown host 0.00.00.00(5432) ===</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:23.616: main pid 2602100: LOG: Restart all children</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 02:30:23.619: main pid 2602100: LOG: execute command: FAIOVER_SCRIPT.sh</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:05:03.346: main pid 2603109: LOG: reaper handler</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:05:03.347: main pid 2603109: LOG: reaper handler: exiting normally</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:05:03.358: main pid 2603109: LOG: reaper handler</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:05:03.358: main pid 2603109: LOG: reaper handler: exiting normally</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:06:20.922: ITN.TaskScheduler pid 2603744: LOG: received degenerate backend request for node_id: 0 from pid [2603744]</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:06:20.922: ITN.TaskScheduler pid 2603744: LOG: signal_user1_to_parent_with_reason(0)</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:06:20.922: main pid 2603109: LOG: Pgpool-II parent process received SIGUSR1</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:06:20.922: main pid 2603109: LOG: Pgpool-II parent process has received failover request</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:06:20.922: main pid 2603109: LOG: === Starting degeneration. shutdown host 0.0.00.00(5432) ===</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:06:20.925: main pid 2603109: LOG: Restart all children</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
2025-03-12 05:06:20.927: main pid 2603109: LOG: execute command: FAILOVER_SCRIPT.sh</div>
<div id="appendonsend"></div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks,</div>
<div class="elementToProof" style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Susihl...</div>
<hr style="display: inline-block; width: 98%;">
<div dir="ltr" id="divRplyFwdMsg"><span style="font-family: Calibri, sans-serif; font-size: 11pt; color: rgb(0, 0, 0);"><b>From:</b> Bo Peng <pengbo@sraoss.co.jp><br>
<b>Sent:</b> Friday, March 14, 2025 4:15 AM<br>
<b>To:</b> Sushil Shirodkar <sushilps@hotmail.com><br>
<b>Cc:</b> pgpool-general@pgpool.net <pgpool-general@pgpool.net><br>
<b>Subject:</b> Re: [pgpool-general: 9393] Pgpool - Failing over from Primary to Secondary for unknown reason.</span>
<div> </div>
</div>
<div style="font-size: 11pt;">Hi,<br>
<br>
Thank you for sharing the logs.<br>
<br>
The logs don't provide any cause for the failover.<br>
Could you share more logs from other processes before the failover?"<br>
Do you see any health check errors in the logs?<br>
<br>
On Wed, 12 Mar 2025 08:01:15 +0000<br>
Sushil Shirodkar <sushilps@hotmail.com> wrote:<br>
<br>
> Hi there,<br>
><br>
> Here is the info. Related to PID 2600121.<br>
> -------------------------------------------------------------------------------------------------------------------<br>
> 2025-03-12 01:10:27.296: child pid 2600121: LOG: new connection received<br>
> 2025-03-12 01:10:27.296: child pid 2600121: DETAIL: connecting host=0.0.00.00 port=51121<br>
><br>
> 2025-03-12 01:39:34.138: IS.Server pid 2600121: LOG: received degenerate backend request for node_id: 0 from pid [2600121]<br>
> 2025-03-12 01:39:34.138: IS.Server pid 2600121: LOG: signal_user1_to_parent_with_reason(0)<br>
><br>
> 2025-03-12 01:39:34.138: main pid 2598786: LOG: Pgpool-II parent process received SIGUSR1<br>
> 2025-03-12 01:39:34.138: main pid 2598786: LOG: Pgpool-II parent process has received failover request<br>
> 2025-03-12 01:39:34.138: main pid 2598786: LOG: === Starting degeneration. shutdown host 0.0.00.00(5432) ===<br>
> 2025-03-12 01:39:34.141: main pid 2598786: LOG: Restart all children<br>
> 2025-03-12 01:39:34.142: main pid 2598786: LOG: execute command: FAILOVER_SCRIPT.sh<br>
> --------------------------------------------------------------------------------------------------------------------------<br>
><br>
> It happened again basically 4th time. This time I have shutdown/started all the 3 instances. Will see if that helps.<br>
><br>
><br>
> Thanks,<br>
> Sushil...<br>
> ________________________________<br>
> From: Bo Peng <pengbo@sraoss.co.jp><br>
> Sent: Wednesday, March 12, 2025 2:05 AM<br>
> To: Sushil Shirodkar <sushilps@hotmail.com><br>
> Cc: pgpool-general@pgpool.net <pgpool-general@pgpool.net><br>
> Subject: Re: [pgpool-general: 9393] Pgpool - Failing over from Primary to Secondary for unknown reason.<br>
><br>
> Hi,<br>
><br>
> Thank you for sharing the logs.<br>
><br>
> The information related to failover was not output in these logs.<br>
> Could you share more logs related to the process with PID 2600121?<br>
><br>
> On Wed, 12 Mar 2025 03:57:56 +0000<br>
> Sushil Shirodkar <sushilps@hotmail.com> wrote:<br>
><br>
> > The following appeared before the failover, nothing abnormal. Similar lines in previous 1 hour of log file.<br>
> ><br>
> > 2025-03-12 01:35:01.745: child pid 2599666: LOG: new connection received<br>
> > 2025-03-12 01:35:01.745: child pid 2599666: DETAIL: connecting host=[local]<br>
> > 2025-03-12 01:35:01.807: child pid 2600001: LOG: new connection received<br>
> > 2025-03-12 01:35:01.808: child pid 2600001: DETAIL: connecting host=[local]<br>
> > 2025-03-12 01:35:03.000: main pid 2598786: LOG: reaper handler<br>
> > 2025-03-12 01:35:03.001: main pid 2598786: LOG: reaper handler: exiting normally<br>
> > ________________________________<br>
> > From: Bo Peng <pengbo@sraoss.co.jp><br>
> > Sent: Tuesday, March 11, 2025 11:42 PM<br>
> > To: Sushil Shirodkar <sushilps@hotmail.com><br>
> > Cc: pgpool-general@pgpool.net <pgpool-general@pgpool.net><br>
> > Subject: Re: [pgpool-general: 9393] Pgpool - Failing over from Primary to Secondary for unknown reason.<br>
> ><br>
> > Hi,<br>
> ><br>
> > To figure out the cause of the failover, please check the<br>
> > Pgpool-II logs right before the failover occurred.<br>
> > Did any error occur?<br>
> ><br>
> > On Wed, 12 Mar 2025 03:22:45 +0000<br>
> > Sushil Shirodkar <sushilps@hotmail.com> wrote:<br>
> ><br>
> > ><br>
> > > Hello all,<br>
> > ><br>
> > > We have 3 servers env. Pgpool + Master/Slave on AWS, for some reason Pgpool thinks Master is down and promotes Slave as Read/Write. Log file on Pgpool shows the below mentioned message during the failover. We had failure thrice in a single day so
far today on this production env. with diff. pids, nothing changed in last couple of months.<br>
> > ><br>
> > > TS_SERVER pid 2600121 LOG-received degenerate backend request for node_id- 0 from pid 2600121<br>
> > ><br>
> > > Env:<br>
> > > Pgpool-II => 4.5.2<br>
> > > Postgres => 16.4<br>
> > > Ubuntu => 22.04.4<br>
> > ><br>
> > > Would be helpful if you folks share your thoughts on the above.<br>
> > ><br>
> > ><br>
> > > Thanks,<br>
> > > Sushil....<br>
> ><br>
> ><br>
> > --<br>
> > Bo Peng <pengbo@sraoss.co.jp><br>
> > SRA OSS K.K.<br>
> > TEL: 03-5979-2701 FAX: 03-5979-2702<br>
> > URL: <a href="https://www.sraoss.co.jp/" id="OWA4fe54d04-650a-4bb6-a6a7-98ae80808d90" class="OWAAutoLink" data-auth="NotApplicable">
https://www.sraoss.co.jp/</a><br>
><br>
><br>
> --<br>
> Bo Peng <pengbo@sraoss.co.jp><br>
> SRA OSS K.K.<br>
> TEL: 03-5979-2701 FAX: 03-5979-2702<br>
> URL: <a href="https://www.sraoss.co.jp/" id="OWA1c494518-0519-f0aa-ed79-aeea23a2fe2f" class="OWAAutoLink" data-auth="NotApplicable">
https://www.sraoss.co.jp/</a><br>
<br>
<br>
--<br>
Bo Peng <pengbo@sraoss.co.jp><br>
SRA OSS K.K.<br>
TEL: 03-5979-2701 FAX: 03-5979-2702<br>
URL: <a href="https://www.sraoss.co.jp/" id="OWAa83976d8-8765-6a66-ab4c-eb32ab024555" class="OWAAutoLink" data-auth="NotApplicable">
https://www.sraoss.co.jp/</a></div>
</body>
</html>