[pgpool-general: 7571] Re: Resource temporarily unavailable

[Tatsuo Ishii]

> Hello Tatsuo,
> This is the result:
> Limit                     Soft Limit           Hard Limit           Units
> Max cpu time              unlimited            unlimited            seconds
> Max file size             unlimited            unlimited            bytes
> Max data size             unlimited            unlimited            bytes
> Max stack size            8388608              unlimited            bytes
> Max core file size        0                    unlimited            bytes
> Max resident set          unlimited            unlimited            bytes
> Max processes             514179               514179               processes
> Max open files            1024                 4096                 files
> Max locked memory         65536                65536                bytes
> Max address space         unlimited            unlimited            bytes
> Max file locks            unlimited            unlimited            locks
> Max pending signals       514179               514179               signals
> Max msgqueue size         819200               819200               bytes
> Max nice priority         0                    0
> Max realtime priority     0                    0
> Max realtime timeout      unlimited            unlimited            us
> 
> Pgpool service starts without any problems and creates about 1008 tasks - problem apprears after some time. Today it happened again:
> 
> pgs00000161:~ # systemctl status pgpool-II-11.service
> ● pgpool-II-11.service - PGPool-II Middleware Between PostgreSQL Servers And PostgreSQL Database Clients
>    Loaded: loaded (/usr/lib/systemd/system/pgpool-II-11.service; disabled; vendor preset: disabled)
>    Active: active (running) since Mon 2021-05-10 16:15:04 CEST; 17h ago
>   Process: 20750 ExecStop=/usr/pgpool-11/bin/pgpool -f /etc/pgpool-II-11/pgpool.conf -m fast stop (code=killed, signal=INT)
>  Main PID: 21255 (pgpool)
>     Tasks: 3 (limit: 1024)
>    CGroup: /system.slice/pgpool-II-11.service
>            ├─21255 /usr/pgpool-11/bin/pgpool -f /etc/pgpool-II-11/pgpool.conf -n -D
>            ├─22359 pgpool: health check process(0)
>            └─22360 pgpool: health check process(1)
> 
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2906-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14222
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2907-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 7431 exits wi...tus 256
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2908-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14223
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2909-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 8142 exits wi...tus 256
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2910-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14224
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2911-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 9498 exits wi...tus 256
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2912-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14225
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2913-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 4782 exits wi...tus 256
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2914-1] 2021-05-10 23:36:38: pid 21255: FATAL:  failed to fork a child
> May 10 23:36:38 pgs00000161 pgpool[21255]: [2914-2] 2021-05-10 23:36:38: pid 21255: DETAIL:  system call fork() failed with rea...ailable
> Hint: Some lines were ellipsized, use -l to show in full.
> 
> It looks like processes responsible for client connections disappear suddenly and new instances cannot be created but restart of pgpool service helps and it works well again some time.
> I have recommendation to set TasksMax to "infinity" or value much higher than num_init_children*max_pool, but i will be able to verify it during service window in next six hours.

Another possibility is number of file descriptors. On my Lunux box
after startinf 3-node Pgpool-II with 3-node PostgreSQL, I see:

$ cat /proc/sys/fs/file-nr
21700	0	1612809

So 21700 file descriptors out of 1612809 are used. Do you see number
of the used fds are closing to the max file descriptor on your system?

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: http://www.sraoss.co.jp/index_en.php
Japanese:http://www.sraoss.co.jp