[pgpool-general: 7565] Re: Resource temporarily unavailable

Tue May 11 16:50:58 JST 2021

Hello Tatsuo,
This is the result:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             514179               514179               processes
Max open files            1024                 4096                 files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       514179               514179               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

Pgpool service starts without any problems and creates about 1008 tasks - problem apprears after some time. Today it happened again:

pgs00000161:~ # systemctl status pgpool-II-11.service
● pgpool-II-11.service - PGPool-II Middleware Between PostgreSQL Servers And PostgreSQL Database Clients
   Loaded: loaded (/usr/lib/systemd/system/pgpool-II-11.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2021-05-10 16:15:04 CEST; 17h ago
  Process: 20750 ExecStop=/usr/pgpool-11/bin/pgpool -f /etc/pgpool-II-11/pgpool.conf -m fast stop (code=killed, signal=INT)
 Main PID: 21255 (pgpool)
    Tasks: 3 (limit: 1024)
   CGroup: /system.slice/pgpool-II-11.service
           ├─21255 /usr/pgpool-11/bin/pgpool -f /etc/pgpool-II-11/pgpool.conf -n -D
           ├─22359 pgpool: health check process(0)
           └─22360 pgpool: health check process(1)

May 10 23:36:38 pgs00000161 pgpool[21255]: [2906-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14222
May 10 23:36:38 pgs00000161 pgpool[21255]: [2907-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 7431 exits wi...tus 256
May 10 23:36:38 pgs00000161 pgpool[21255]: [2908-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14223
May 10 23:36:38 pgs00000161 pgpool[21255]: [2909-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 8142 exits wi...tus 256
May 10 23:36:38 pgs00000161 pgpool[21255]: [2910-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14224
May 10 23:36:38 pgs00000161 pgpool[21255]: [2911-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 9498 exits wi...tus 256
May 10 23:36:38 pgs00000161 pgpool[21255]: [2912-1] 2021-05-10 23:36:38: pid 21255: LOG:  fork a new child process with pid: 14225
May 10 23:36:38 pgs00000161 pgpool[21255]: [2913-1] 2021-05-10 23:36:38: pid 21255: LOG:  child process with pid: 4782 exits wi...tus 256
May 10 23:36:38 pgs00000161 pgpool[21255]: [2914-1] 2021-05-10 23:36:38: pid 21255: FATAL:  failed to fork a child
May 10 23:36:38 pgs00000161 pgpool[21255]: [2914-2] 2021-05-10 23:36:38: pid 21255: DETAIL:  system call fork() failed with rea...ailable
Hint: Some lines were ellipsized, use -l to show in full.

It looks like processes responsible for client connections disappear suddenly and new instances cannot be created but restart of pgpool service helps and it works well again some time.
I have recommendation to set TasksMax to "infinity" or value much higher than num_init_children*max_pool, but i will be able to verify it during service window in next six hours.
Best regards

Andrzej Rozmus
Starszy Technolog
Departament Systemów Administracji Rządowej
Dział Ministerstwo Finansów
Zespół Projektowy AIS/AES
Asseco Poland S.A.
ul. Branickiego 13
02-972 Warszawa
tel. kom. +48 502 270 059
andrzej.rozmus at asseco.pl

-----Original Message-----
From: Tatsuo Ishii <ishii at sraoss.co.jp> 
Sent: Tuesday, May 11, 2021 7:03 AM
To: Rozmus Andrzej <Andrzej.Rozmus at asseco.pl>
Cc: pgpool-general at pgpool.net; Wierzbicki Sebastian <sebastian.wierzbicki at asseco.pl>; tomasz.pajda at mf.gov.pl
Subject: Re: [pgpool-general: 7563] Resource temporarily unavailable

> Hello Tatsuo,
> We have encountered such a problem today:
> FATAL:  failed to fork a child
> DETAIL:  system call fork() failed with reason: Resource temporarily 
> unavailable It seems to be strange because all necessary resources (CPU, RAM) should be available, number of allowed proccesses that can be created by postgres user (pgpool is registered in systemd and works with postgres account) is definitely enough to cover 2048 processes (1024 processe of pgpool and 1024 processes of postgresql). Pgpool hanged with only two subprocesses started:
> 
> Main PID: 9584 (pgpool)
>     Tasks: 3 (limit: 1024)
>    CGroup: /system.slice/pgpool-II-11.service
>            ├─ 9584 /usr/pgpool-11/bin/pgpool -f /etc/pgpool-II-11/pgpool.conf -n -D
>            ├─10596 pgpool: health check process(0)
>            └─10597 pgpool: health check process(1) Do you have any 
> suggestions what could cause this problem? We have changed „TasksMax” option from default 512 to 1024 for postgresql and pgpool services but as i wrote before – number of processes that can be created by postgres user is far above 2048. I have also checked typical number of running processes in our system (about 500) to exclude problem with insufficient value of kernel.pid_max (32768) parameter.

Can you share the result of:

cat /proc/9584/limits

Best regards,
--
Tatsuo Ishii
SRA OSS, Inc. Japan
English: https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en.php&data=04%7C01%7CAndrzej.Rozmus%40asseco.pl%7C9080bd0cf7d3409a0ffd08d9143a17b7%7C88152bdecfa34a5cb981a785c624bb42%7C0%7C0%7C637563062000719691%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZWKEvIIxp10vwglOHMidLVINHpVR1wx8vaNdn1lr%2FNg%3D&reserved=0
Japanese:https://eur02.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.sraoss.co.jp%2F&data=04%7C01%7CAndrzej.Rozmus%40asseco.pl%7C9080bd0cf7d3409a0ffd08d9143a17b7%7C88152bdecfa34a5cb981a785c624bb42%7C0%7C0%7C637563062000719691%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=kqqZ5qS5eUNwfRmfN3dwisIjFhEgXWQjTY%2Bnd2rbjmw%3D&reserved=0