<br><br><div class="gmail_quote">2013/3/4 Tatsuo Ishii <span dir="ltr"><<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="im">> Hi Tatsuo, as promised I've tested pgpool 3.2.3 on AIX 5.2<br>
><br>
> it behaves in a really strange way, at the beginning when I was starting<br>
> pgpool it was seeming to be working pretty fine,<br>
> after some stop and start it became to behave really strangely.<br>
><br>
> it seems to mess with some cached status does exists some cache_status<br>
> file? i removed backend files .s.PGSQL.9898 % .s.PGSQL.9999 butit doesn't<br>
> change anything.<br>
<br>
</div>I don't think so. The cached status file is located at<br>
/tmp/pgpool_status in your case. From the log, it says:<br>
<br>
2013-03-04 17:44:18 ERROR: pid 72000: Could not read backend status file as /tmp/pgpool_status. reason: No such file or directory<br>
<br>
So pgpool did not read cached status and this is normal. (BTW, if you<br>
want to be sure to ignore the status file, you could start pgpool with<br>
-D option).<br>
<br><br></blockquote><div> </div><div>Yes the things you say are right, the strange thing is that the /tmp/pgpool_status file exists but pgpool is not able to read it, it's strange cause user has right to read and write on it. any way after that I started using the -D option that file has been discarded automatically by pgpool process but this didn't help.<br>
</div><div>My suspect of a strange behaviour cause of some kind of cached status comes from the fact that in the very beginning, when I was having the same configuration that I'm having now all was working really fine.<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
>From the log file I noticed to the health check function failed to<br>
connect to backend 1. Are you sure that backend1(devlam0) allows to<br>
connect with user = postgres without password from the where pgpool is<br>
running on?<br></blockquote><div><br>Yes backend1(devlam0) allows to connect normally. It was down in that very moment when I launched pgpool in debug mode, <br>I get this trouble only when a failover happen, but pgpool seems not to be able to manage correctly the failover and the main process go into defunct status.<br>
<br>I repeat again that earlier than yesterday it was working pretty fine only a few times I was getting strange behaviours but i was not analizing properly processes in those moments, <br>the only thing that I changed yesterday is that I've installed the pgpool-recovery module compiled from the source tarball but I do not think that this could make pgpool unstable by logic it should be used only when calling<br>
the pcp_recovery_node command.<br><br>Sorry if I didn't wrote all these details before.<br><br>to let you understand better I attach the debug log and I will list the ps commands related to this scenario:<br><br>1) both backends working at the pgpool startup.<br>
<br>2) simulating the node1 failover sending the stop command to postgres on devlam0 server.<br><br>this is the process list sequence<br><br>the failover simulation time is at 2013-03-05 18:32:10<br><br><br>process list before failover<br>
<br>[postgres@bldlam:/home/postgres]# ps -fu postgres | grep pgpool<br>postgres 7580 123522 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>postgres 44862 50512 1 18:30:32 pts/3 0:00 grep pgpool <br>
postgres 48162 123522 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>postgres 49986 123522 0 18:29:46 pts/1 0:00 pgpool: PCP: wait for connection request <br>postgres 83912 123522 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>
postgres 84228 123522 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>postgres 104292 123522 0 18:29:46 pts/1 0:00 pgpool: worker process <br>postgres 104468 123522 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>
postgres 116214 123522 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>postgres 123522 25022 0 18:29:46 pts/1 0:00 bin/pgpool -d -D -n <br><br>process list after failover<br><br>[postgres@bldlam:/home/postgres]# ps -fu postgres | grep pgpool<br>
postgres 7580 1 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>postgres 41166 50512 1 18:34:05 pts/3 0:00 grep pgpool <br>postgres 48162 1 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>
postgres 49986 1 0 18:29:46 pts/1 0:00 pgpool: PCP: wait for connection request <br>postgres 83912 1 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>postgres 84228 1 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>
postgres 104292 1 0 18:29:46 pts/1 0:00 pgpool: worker process <br>postgres 104468 1 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>postgres 116214 1 0 18:29:46 pts/1 0:00 pgpool: wait for connection request <br>
<br><br>--<br>Daniele Di Vito<br><br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">--<br>
Tatsuo Ishii<br>
SRA OSS, Inc. Japan<br>
English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
> No changes in pgpool.conf nor in postgresql.conf<br>
> it returned to loose communication with the main process as described for<br>
> 3.2.2 version, the main process abnormally exit<br>
><br>
> I've tried to change some settings in pgpool.conf but it doesn't take<br>
> effets on pool behaviour<br>
><br>
> For example I tried to increase child life time cause i was suspecting some<br>
> issue on child process destruction<br>
><br>
> child_life_time = 100000000000<br>
> # Pool exits after being idle for this<br>
> many seconds<br>
> child_max_connections = 0<br>
> # Pool exits after receiving that many<br>
> connections<br>
> # 0 means no exit<br>
> connection_life_time = 0<br>
> # Connection to backend closes after<br>
> being idle for this many seconds<br>
> # 0 means no close<br>
> client_idle_limit = 0<br>
><br>
><br>
> I attach debug output truss output and pgpool.conf<br>
><br>
> Hoping that someone can help me to solve this trouble on AIX.<br>
><br>
> Thanks<br>
><br>
> Great!<br>
>><br>
>> Yes, I've just install the 3.2.3 et it seems to be working great!<br>
>><br>
>> Obviouslly I will test it properly and I will let you know how this<br>
>> version works on AIX.<br>
>><br>
>> Thanks again<br>
>><br>
>><br>
>><br>
>> 2013/2/22 Tatsuo Ishii <<a href="mailto:ishii@postgresql.org">ishii@postgresql.org</a>><br>
>><br>
>>> I'm going to check the data you posted.<br>
>>><br>
>>> In the meatime, I think it is posiible your problem is caused by the<br>
>>> bug fixed in pgpool-II 3.2.3, especiall if the problem goes away by<br>
>>> disabling ealth checking. Can you try 3.2.3?<br>
>>> --<br>
>>> Tatsuo Ishii<br>
>>> SRA OSS, Inc. Japan<br>
>>> English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>>> Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
>>><br>
>>> > Thanks for your quick reply Tatsuo,<br>
>>> ><br>
>>> > Before to get your reply i tried to understand where the main process<br>
>>> stops<br>
>>> > adding some print in the line right above every exit call in the main.c<br>
>>> > class.<br>
>>> > I found out that it never stops calling the exit command in an explicit<br>
>>> > way.<br>
>>> ><br>
>>> > After your reply I tracked it's behaviour using truss.<br>
>>> > Truss output is in the attched file.<br>
>>> ><br>
>>> ><br>
>>> > Commenting the dup2 file in the demonize function, I'd been able to get<br>
>>> > which error happens just before the process "death".<br>
>>> ><br>
>>> > This is the error:<br>
>>> ><br>
>>> > pool_flush_it: write failed to backend (0). reason: Socket is not<br>
>>> connected<br>
>>> > offset: 0 wlen: 41<br>
>>> ><br>
>>> > it happens after the first health_check call.<br>
>>> ><br>
>>> > it seems that the socket is not connected to the local backend (backend<br>
>>> 0<br>
>>> > is on the same host where pgpool is running) but stragely the<br>
>>> replication<br>
>>> > on the backend 0 it work normally so I think that it's connected.<br>
>>> ><br>
>>> > The real trouble is that pgppol will never check for failover or<br>
>>> failback<br>
>>> > loosing the main process.<br>
>>> ><br>
>>> > The pgsql version of every backend is 8.3. I attach the pgpool config<br>
>>> file<br>
>>> > too.<br>
>>> ><br>
>>> ><br>
>>> > Thanks again for your help<br>
>>> ><br>
>>> ><br>
>>> > --<br>
>>> ><br>
>>> > Daniele Di vito<br>
>>> ><br>
>>> ><br>
>>> > 2013/2/21 Tatsuo Ishii <<a href="mailto:ishii@postgresql.org">ishii@postgresql.org</a>><br>
>>> ><br>
>>> >> > HI everybody, I've compiled pgpool 3.2.2 on AIX 5.2.<br>
>>> >> ><br>
>>> >> > I configured the pool for using replication mode. The configuration<br>
>>> is<br>
>>> >> > working really fine on some linux virtual machine, but when I try to<br>
>>> use<br>
>>> >> > pgpool with the same configuration on AIX I have a big trouble.<br>
>>> >> ><br>
>>> >> > Starting with "pgpool -d" the server seems to be starting normally.<br>
>>> it<br>
>>> >> > create pcp process and it create the pool connections waiting for<br>
>>> >> > connection requests.<br>
>>> >> ><br>
>>> >> > When I lunch a "ps -fu postgres | grep pgpool" i get this output:<br>
>>> >> ><br>
>>> >> ><br>
>>> >> ><br>
>>> >> > postgres 62164 1 0 10:05:48 - 0:00 pgpool: wait for<br>
>>> >> > connection request<br>
>>> >> > postgres 75470 1 0 10:05:48 - 0:00 pgpool: PCP: wait<br>
>>> for<br>
>>> >> > connection request<br>
>>> >> > postgres 84072 1 0 10:05:48 - 0:00 pgpool: wait for<br>
>>> >> > connection request<br>
>>> >> > postgres 96828 1 0 10:05:48 - 0:00 pgpool: wait for<br>
>>> >> > connection request<br>
>>> >> > postgres 100026 1 0 10:05:48 - 0:00 pgpool: wait for<br>
>>> >> > connection request<br>
>>> >> > postgres 106670 1 0 10:05:47 - 0:00 pgpool: wait for<br>
>>> >> > connection request<br>
>>> >> > postgres 109864 1 0 10:05:48 - 0:00 pgpool: worker<br>
>>> process<br>
>>> >> > postgres 116412 1 0 10:05:48 - 0:00 pgpool: wait for<br>
>>> >> > connection request<br>
>>> >> ><br>
>>> >> > but, as you can see looking at the output listed above,no pgpool<br>
>>> daemon<br>
>>> >> is<br>
>>> >> > running and every subprocess created by it now have as ppid 1.<br>
>>> >> ><br>
>>> >> > if I look into the pgpool.pid i get a pid that is not running on the<br>
>>> AIX<br>
>>> >> > machine.<br>
>>> >> > Obviously if i try to stop pgpool it says that the process is not<br>
>>> >> running<br>
>>> >> > so i have to kill every process and to remove every temporary file<br>
>>> >> manually.<br>
>>> >> ><br>
>>> >> > If i run it without a daemon using "pgppool -n"<br>
>>> >> ><br>
>>> >> > the pgpool -n process is listed for some minutes in the "ps -fu<br>
>>> >> postgres |<br>
>>> >> > grep pgpool" and every subprocess have the right ppid.<br>
>>> >> > Some minutes later i get the same output I listed for the "pgpool -d"<br>
>>> >> > command start.<br>
>>> >> ><br>
>>> >> > Any idea on how to solve this trouble?<br>
>>> >> ><br>
>>> >> > I've already tried to find some error while in debug mode, but no<br>
>>> error<br>
>>> >> > listed.<br>
>>> >><br>
>>> >> Does AIX have something like "strace" or "truss"? If so, taking a<br>
>>> >> system call trace by using it, may provide valuable information. You<br>
>>> >> take system call trace until pgpool-II parent process disappears.<br>
>>> >> --<br>
>>> >> Tatsuo Ishii<br>
>>> >> SRA OSS, Inc. Japan<br>
>>> >> English: <a href="http://www.sraoss.co.jp/index_en.php" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
>>> >> Japanese: <a href="http://www.sraoss.co.jp" target="_blank">http://www.sraoss.co.jp</a><br>
>>> >><br>
>>><br>
>><br>
>><br>
</div></div></blockquote></div><br>