[pgpool-hackers: 4046] Re: [pgpool-general: 7811] Re: Error: sorry, too many clients

[Tatsuo Ishii]

We have gotten multiple complains like below.

I still have not gotten good idea to fix the counter leak in any case,
but I feel like I need to fix the case at least when
reserved_connections is 0.

Varying no objection, I will commit the patch to all supported
branches.
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp

>>> In our environment:
>>> reserved_connections=0
>>> listen_backlog_multiplier=2
>>> 
>>> So, if I understand correctly, we have 2 different problematic cases:
>>> 
>>> (1) a client silently disappears (doesn't properly close the connection):
>>> In this case, setting `child_idle_limit` to a non-zero value (say 15min)
>>> could mitigate the problem, and eventually the child worker will again be
>>> available to handle an incoming connection.
>>>
>>> (2) a server-side child process crashes: In this case, Pgpool still keeps a
>>> dangling reference to this process, and if some client is routed there will
>>> get a "sorry, too many clients" error.
>> 
>> Since the counter is shared by all process, any process could get a
>> "sorry..." error.
>> 
>>> As for now, this problem cannot be
>>> mitigated unless the (parent) Pgpool process is restarted.
>> 
>> Yes.
>> 
>>> One more question:
>>> What can make a child process crash? (well, I mean, what are the most
>>> common reasons for that).
>> 
>> Besides admin's mistake, probabbly the most case is OOM killer.
>> 
>>> Does this crash leave any trace (that I should search for) in Pgpool's
>>> error log? Does parent Pgpool get notified on this (e.g. via waitpid or
>>> SIGCHLD)?
>> 
>> Yes, Pgpool parent process should leave a log when catching SIGCHLD.
>> If I kill a child process using kill -9, I got following log:
>> 
>> 2021-10-18 14:44:48.951: main pid 997555: LOG:  child process with pid: 997582 exits with status 9 by signal 9
>> 2021-10-18 14:44:48.952: main pid 997555: LOG:  fork a new child process with pid: 997609
> 
> Actually if reserved_connections=0, then there's no need to manage the
> connection counter. So I have created a patch to eliminate the
> management part. You will never be troubled by the "sorry..."  error.
> 
> If you like, please try attached patch.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp