<div dir="ltr">Could this error happen if "connection_cache = off" ?<div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On 23 February 2016 at 12:27, Muhammad Usama <span dir="ltr"><<a href="mailto:m.usama@gmail.com" target="_blank">m.usama@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Tue, Feb 23, 2016 at 4:16 AM, Tatsuo Ishii <<a href="mailto:ishii@postgresql.org">ishii@postgresql.org</a>> wrote:<br>
> Usama,<br>
><br>
> Doesn't pgpool-II 3.1 have the same problem?<br>
<br>
</span>Sorry, I missed that and 3_0 aswell. I have pushed the same change to<br>
the both branches.<br>
<br>
Regards<br>
<span class="HOEnZb"><font color="#888888">Muhammad Usama<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
<br>
><br>
> Best regards,<br>
> --<br>
> Tatsuo Ishii<br>
> SRA OSS, Inc. Japan<br>
> English: <a href="http://www.sraoss.co.jp/index_en.php" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en.php</a><br>
> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
><br>
>> Hi<br>
>><br>
>> I have pushed the above fix in all branches from pgpool-II 3.2 onwards.<br>
>> <a href="http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6afacb1b19603b37e3d005963182258b9f4fca49" rel="noreferrer" target="_blank">http://git.postgresql.org/gitweb?p=pgpool2.git;a=commitdiff;h=6afacb1b19603b37e3d005963182258b9f4fca49</a><br>
>><br>
>> Thanks again for your help in verifying and testing the fix.<br>
>><br>
>> Kind regards<br>
>> Muhammad Usama<br>
>><br>
>><br>
>> On Sat, Feb 20, 2016 at 1:03 AM, Muhammad Usama <<a href="mailto:m.usama@gmail.com">m.usama@gmail.com</a>> wrote:<br>
>><br>
>>> Hi<br>
>>><br>
>>> Many thanks for the confirmation, the fix needs to go in all branches and<br>
>>> I will push it in the morning.<br>
>>><br>
>>> Regards<br>
>>> Muhammad Usama<br>
>>><br>
>>> Sent from my iPhone<br>
>>><br>
>>> > On 19-Feb-2016, at 11:52 PM, Gerhard Wiesinger <<a href="mailto:lists@wiesinger.com">lists@wiesinger.com</a>><br>
>>> wrote:<br>
>>> ><br>
>>> > Hello,<br>
>>> ><br>
>>> > Can confirm that this patch worked for me (tested the 3.5 patch<br>
>>> version), nearly 2 days without any problem. Can you please add it to the<br>
>>> git repo and make a new release (3.4, 3.5).<br>
>>> ><br>
>>> > Thnx.<br>
>>> ><br>
>>> > Ciao,<br>
>>> > Gerhard<br>
>>> ><br>
>>> ><br>
>>> >> On 16.02.2016 11:44, Muhammad Usama wrote:<br>
>>> >> Hi<br>
>>> >><br>
>>> >> Many thanks for the reply and a good news that you are not getting<br>
>>> >> stuck connection issue after the patch.<br>
>>> >><br>
>>> >> Thanks<br>
>>> >> Best regards<br>
>>> >> Muhammad Usama<br>
>>> >><br>
>>> >><br>
>>> >>> On Fri, Feb 12, 2016 at 9:45 PM, Paweł Ufnalewski <<a href="mailto:archon@foap.com">archon@foap.com</a>><br>
>>> wrote:<br>
>>> >>> Hmm it looks like it's fine now. Right now I only see these in log:<br>
>>> >>><br>
>>> >>> 2016-02-12 17:28:12: pid 8838: LOG: child process with pid: 27299<br>
>>> exits<br>
>>> >>> with status 256<br>
>>> >>> 2016-02-12 17:28:12: pid 8838: LOG: fork a new child process with<br>
>>> pid: 6140<br>
>>> >>> 2016-02-12 17:30:42: pid 8838: LOG: child process with pid: 5571<br>
>>> exits with<br>
>>> >>> status 512<br>
>>> >>> 2016-02-12 17:30:42: pid 8838: LOG: fork a new child process with<br>
>>> pid: 6720<br>
>>> >>> 2016-02-12 17:30:43: pid 8838: LOG: child process with pid: 4444<br>
>>> exits with<br>
>>> >>> status 512<br>
>>> >>> 2016-02-12 17:30:43: pid 8838: LOG: fork a new child process with<br>
>>> pid: 6751<br>
>>> >>> 2016-02-12 17:35:42: pid 8838: LOG: child process with pid: 6140<br>
>>> exits with<br>
>>> >>> status 512<br>
>>> >>> 2016-02-12 17:35:42: pid 8838: LOG: fork a new child process with<br>
>>> pid: 7868<br>
>>> >>> 2016-02-12 17:40:42: pid 8838: LOG: child process with pid: 6751<br>
>>> exits with<br>
>>> >>> status 512<br>
>>> >>> 2016-02-12 17:40:42: pid 8838: LOG: fork a new child process with<br>
>>> pid: 9018<br>
>>> >>> 2016-02-12 17:40:42: pid 8838: LOG: child process with pid: 6720<br>
>>> exits with<br>
>>> >>> status 512<br>
>>> >>> 2016-02-12 17:40:42: pid 8838: LOG: fork a new child process with<br>
>>> pid: 9019<br>
>>> >>><br>
>>> >>> Thank you!<br>
>>> >>><br>
>>> >>> Best regards,<br>
>>> >>> Paweł Ufnalewski<br>
>>> >>> Infrastructure Architect at Foap.com<br>
>>> >>><br>
>>> >>> W dniu 2016-02-09 o 14:02, Muhammad Usama pisze:<br>
>>> >>><br>
>>> >>>> Hi<br>
>>> >>>><br>
>>> >>>> Many thanks for sharing the pgpool.log, The log shared by you does<br>
>>> >>>> contains some error messages "ERROR: unable to to flush data to<br>
>>> >>>> frontend" that have the potential to cause the stuck connection<br>
>>> >>>> Can you please try out the attached patch if it fix the problem. I am<br>
>>> >>>> attaching the patches for both 3_5 and 3_4 branches, please use the<br>
>>> >>>> respective patch as per your setup. Hopefully this should fix the<br>
>>> >>>> stuck issue.<br>
>>> >>>><br>
>>> >>>> Kind regards<br>
>>> >>>> Muhammad Usama<br>
>>> >>>><br>
>>> >>>><br>
>>> >>>><br>
>>> >>>>> On Mon, Feb 8, 2016 at 8:49 PM, Paweł Ufnalewski <<a href="mailto:archon@foap.com">archon@foap.com</a>><br>
>>> wrote:<br>
>>> >>>>> Hi,<br>
>>> >>>>><br>
>>> >>>>> It looks like it hangs in this places (see attachment). Problem<br>
>>> is,<br>
>>> >>>>> that<br>
>>> >>>>> developer responsible for app has changed something in code, so<br>
>>> >>>>> connections<br>
>>> >>>>> now closes properly from client side (before I got a lot of these<br>
>>> errors:<br>
>>> >>>>><br>
>>> >>>>> 2016-02-08 09:33:39: pid 8472: ERROR: unable to read data from<br>
>>> frontend<br>
>>> >>>>> 2016-02-08 09:33:39: pid 8472: DETAIL: EOF encountered with<br>
>>> frontend)<br>
>>> >>>>> .<br>
>>> >>>>><br>
>>> >>>>> Best regards,<br>
>>> >>>>> Paweł Ufnalewski<br>
>>> >>>>> Infrastructure Architect at Foap.com<br>
>>> >>>>><br>
>>> >>>>> W dniu 2016-02-08 o 09:00, Muhammad Usama pisze:<br>
>>> >>>>><br>
>>> >>>>> Hi<br>
>>> >>>>><br>
>>> >>>>> Thanks in advance for the help. If you could share the pgpool-II log<br>
>>> >>>>> when the stuck connection happens that would help us in identifiny<br>
>>> and<br>
>>> >>>>> rectifing the problem.<br>
>>> >>>>><br>
>>> >>>>> Thanks<br>
>>> >>>>> Best regards<br>
>>> >>>>> Muhammad Usama<br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>> On Mon, Feb 8, 2016 at 11:36 AM, Paweł Ufnalewski <<a href="mailto:archon@foap.com">archon@foap.com</a>><br>
>>> >>>>> wrote:<br>
>>> >>>>><br>
>>> >>>>> Hi,<br>
>>> >>>>><br>
>>> >>>>> just to let you know - I'm having the same problem with 3.4.4<br>
>>> >>>>> version<br>
>>> >>>>> (DISCARD ALL appears slower than in 3.4.3 I think, but it still<br>
>>> does).<br>
>>> >>>>> How<br>
>>> >>>>> can I help to fix this problem?<br>
>>> >>>>><br>
>>> >>>>> Best regards,<br>
>>> >>>>> Paweł Ufnalewski<br>
>>> >>>>> Infrastructure Architect at Foap.com<br>
>>> >>>>><br>
>>> >>>>> W dniu 2016-02-01 o 08:44, Muhammad Usama pisze:<br>
>>> >>>>><br>
>>> >>>>> Hi Gerhard<br>
>>> >>>>><br>
>>> >>>>> Many thanks for testing and pointing this out. It's unfortunate that<br>
>>> you<br>
>>> >>>>> are<br>
>>> >>>>> still getting the stuck connection issue. If it is possible can you<br>
>>> >>>>> please<br>
>>> >>>>> share the pgpool-II log for the time when this stuck connection issue<br>
>>> >>>>> happens. I am more interested in seeing which exact error message<br>
>>> that<br>
>>> >>>>> caused the child process to jump to error handler from where the<br>
>>> child<br>
>>> >>>>> process proceeded to send the DISCARD ALL to backend and eventually<br>
>>> got<br>
>>> >>>>> stuck. Since after many tries we are not able to reproduce this<br>
>>> issue, so<br>
>>> >>>>> log would be really helpful in understanding and fixing the problem.<br>
>>> >>>>><br>
>>> >>>>> Best regards<br>
>>> >>>>> Muhammad Usama<br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>> On Sun, Jan 31, 2016 at 9:33 PM, Gerhard Wiesinger <<br>
>>> <a href="mailto:lists@wiesinger.com">lists@wiesinger.com</a>><br>
>>> >>>>> wrote:<br>
>>> >>>>><br>
>>> >>>>> On 28.01.2016 01:10, Tatsuo Ishii wrote:<br>
>>> >>>>><br>
>>> >>>>> On 21.01.2016 20:52, Muhammad Usama wrote:<br>
>>> >>>>><br>
>>> >>>>> Hi<br>
>>> >>>>><br>
>>> >>>>> I am looking into this issue. and unfortunately like Ishii-San I am<br>
>>> >>>>> also not able to reproduce it. But I found one issue in 3.4 that<br>
>>> might<br>
>>> >>>>> cause the problem. Can you please try the attached patch if it solves<br>
>>> >>>>> the problem. Also, if the problem still persists, it would be really<br>
>>> >>>>> helpful if you could share the pgpool-II log.<br>
>>> >>>>><br>
>>> >>>>> I looked at the patch but it includes only logging changes and no<br>
>>> >>>>> functional changes. Therefore I didn't test it. Do you expect and<br>
>>> >>>>> behavioral changes to fix it, and why?<br>
>>> >>>>><br>
>>> >>>>> elog() is not only a logging function, but also it plays very<br>
>>> >>>>> important role including exception handling and error treatments in<br>
>>> >>>>> pgpool-II. If you are familiar with PostgreSQL internals, you may<br>
>>> >>>>> notice it (elog() was imported from PostgreSQL source tree).<br>
>>> >>>>><br>
>>> >>>>> Tried version 3.5.0 where the patch is included. Still not working.<br>
>>> See<br>
>>> >>>>> backtrace below.<br>
>>> >>>>><br>
>>> >>>>> Reverting to 3.3.7 which works perfectly.<br>
>>> >>>>><br>
>>> >>>>> Ciao,<br>
>>> >>>>> Gerhard<br>
>>> >>>>><br>
>>> >>>>> (gdb) back<br>
>>> >>>>> #0 0x00007fd87fdb6d43 in __select_nocancel () from /lib64/libc.so.6<br>
>>> >>>>> #1 0x0000564471af16a1 in pool_check_fd (cp=cp@entry=0x564473dfa610)<br>
>>> at<br>
>>> >>>>> protocol/pool_process_query.c:635<br>
>>> >>>>> #2 0x0000564471af1976 in pool_check_fd (cp=cp@entry=0x564473dfa610)<br>
>>> at<br>
>>> >>>>> protocol/pool_process_query.c:657<br>
>>> >>>>> #3 0x0000564471b1f67b in pool_read (cp=0x564473dfa610,<br>
>>> >>>>> buf=buf@entry=0x7ffc1d71bf97, len=len@entry=1) at<br>
>>> utils/pool_stream.c:162<br>
>>> >>>>> #4 0x0000564471af8e6e in read_kind_from_backend<br>
>>> >>>>> (frontend=frontend@entry=0x564473df3e60,<br>
>>> >>>>> backend=backend@entry=0x564473df2e00,<br>
>>> >>>>> decided_kind=decided_kind@entry=0x7ffc1d71c397 "E") at<br>
>>> >>>>> protocol/pool_process_query.c:3234<br>
>>> >>>>> #5 0x0000564471affdc9 in ProcessBackendResponse<br>
>>> >>>>> (frontend=frontend@entry=0x564473df3e60,<br>
>>> >>>>> backend=backend@entry=0x564473df2e00, state=state@entry<br>
>>> =0x7ffc1d71c41c,<br>
>>> >>>>> num_fields=num_fields@entry=0x7ffc1d71c41a) at<br>
>>> >>>>> protocol/pool_proto_modules.c:2356<br>
>>> >>>>> #6 0x0000564471af5b15 in pool_process_query<br>
>>> (frontend=0x564473df3e60,<br>
>>> >>>>> backend=0x564473df2e00, reset_request=reset_request@entry=1) at<br>
>>> >>>>> protocol/pool_process_query.c:302<br>
>>> >>>>> #7 0x0000564471aed98c in backend_cleanup (backend=<optimized out>,<br>
>>> >>>>> frontend_invalid=frontend_invalid@entry=0 '\000',<br>
>>> frontend=0x564471e09e40<br>
>>> >>>>> <child_frontend>)<br>
>>> >>>>> at protocol/child.c:437<br>
>>> >>>>> #8 0x0000564471af0637 in do_child (fds=fds@entry=0x564473dee030) at<br>
>>> >>>>> protocol/child.c:234<br>
>>> >>>>> #9 0x0000564471ace107 in fork_a_child (fds=0x564473dee030, id=8) at<br>
>>> >>>>> main/pgpool_main.c:678<br>
>>> >>>>> #10 0x0000564471aceb6d in reaper () at main/pgpool_main.c:2254<br>
>>> >>>>> #11 0x0000564471ad322b in PgpoolMain (discard_status=<optimized out>,<br>
>>> >>>>> clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:429<br>
>>> >>>>> #12 0x0000564471acc7b1 in main (argc=<optimized out>,<br>
>>> >>>>> argv=0x7ffc1d7219e8)<br>
>>> >>>>> at main/main.c:310<br>
>>> >>>>><br>
>>> >>>>> #1 0x0000564471af16a1 in pool_check_fd (cp=cp@entry=0x564473dfa610)<br>
>>> at<br>
>>> >>>>> protocol/pool_process_query.c:635<br>
>>> >>>>> 635 fds = select(fd+1, &readmask, NULL,<br>
>>> &exceptmask,<br>
>>> >>>>> timeoutp);<br>
>>> >>>>><br>
>>> >>>>> (gdb) print fd<br>
>>> >>>>> $1 = 8<br>
>>> >>>>> (gdb) print readmask<br>
>>> >>>>> $2 = {fds_bits = {256, 0 <repeats 15 times>}}<br>
>>> >>>>> (gdb) print exceptmask<br>
>>> >>>>> $3 = {fds_bits = {256, 0 <repeats 15 times>}}<br>
>>> >>>>> (gdb) print timeoutp<br>
>>> >>>>> $4 = (struct timeval *) 0x0<br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>><br>
>>> >>>>> _______________________________________________<br>
>>> >>>>> pgpool-general mailing list<br>
>>> >>>>> <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
>>> >>>>> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
>>> >> _______________________________________________<br>
>>> >> pgpool-general mailing list<br>
>>> >> <a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
>>> >> <a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
>>> ><br>
>>><br>
_______________________________________________<br>
pgpool-general mailing list<br>
<a href="mailto:pgpool-general@pgpool.net">pgpool-general@pgpool.net</a><br>
<a href="http://www.pgpool.net/mailman/listinfo/pgpool-general" rel="noreferrer" target="_blank">http://www.pgpool.net/mailman/listinfo/pgpool-general</a><br>
</div></div></blockquote></div><br></div>