[pgpool-general: 8476] Re: failed to lock semaphore

[Tatsuo Ishii]

Hi,

> Greetings,
> 
> I've been working at implementing pgPool recently, and got everything
> working fine for the most part, but I keep seeing some WARNING messages in
> pgpool.log, and was wondering if anyone knows how to fix this:
> 
> 2022-10-26 23:58:02.103: sr_check_worker pid 765970: WARNING:  failed to
> lock semaphore
> 2022-10-26 23:58:02.103: sr_check_worker pid 765970: DETAIL:  Invalid
> argument
> 2022-10-26 23:58:02.103: sr_check_worker pid 765970: WARNING:  failed to
> unlock semaphore
> 2022-10-26 23:58:02.103: sr_check_worker pid 765970: DETAIL:  Invalid
> argument
> 
> These warnings appear in the pgpool.log file. but also when making a client
> connection, such as:
> 
> postgres at psql-01:~$ psql -h spsql-vip -p 5432 -U postgres -c "select now(),
> count(*) from pg_stat_activity;"
> WARNING:  XX000: failed to lock semaphore
> DETAIL:  Invalid argument
> LOCATION:  pool_sema.c:129
> WARNING:  XX000: failed to unlock semaphore
> DETAIL:  Invalid argument
> LOCATION:  pool_sema.c:202
> <..>
>               now              | count
> -------------------------------+-------
>  2022-10-26 23:22:57.193342+00 |   276
> (1 row)

Seems pretty unusual.

> My environment is as follow:
> 
> # lsb_release -a
> ==
> No LSB modules are available.
> Distributor ID: Ubuntu
> Description:    Ubuntu 20.04.5 LTS
> Release:        20.04
> Codename:       focal
> ==
> 
> # apt list --installed | grep pgpool
> ==
> libpgpool2/focal-pgdg,now 4.3.3-1.pgdg20.04+1 amd64 [installed]
> pgpool2/focal-pgdg,now 4.3.3-1.pgdg20.04+1 amd64 [installed]
> postgresql-13-pgpool2/focal-pgdg,now 4.3.3-1.pgdg20.04+1 amd64 [installed]
> ==

I regularly use Ubuntu 20 for my Pgpool-II development work. Although
my installation of Pgpool-II is from source code.

> # pgpool --version
> pgpool-II version 4.3.3 (tamahomeboshi)
> 
> $ postgres --version
> postgres (PostgreSQL) 13.8 (Ubuntu 13.8-1.pgdg20.04+1)
> 
> And lastly, in case it can help. here's a strace snippet which is related
> to this error I believe:
> 
> semtimedop(1, [{7, -1, SEM_UNDO}], 1, NULL) = -1 EINVAL (Invalid argument)
> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
> write(2, "\0\0\245\0NI\0\0t2022-10-27 13:06:21.879"..., 174) = 174
> semtimedop(1, [{7, 1, SEM_UNDO}], 1, NULL) = -1 EINVAL (Invalid argument)
> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
> stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=118, ...}) = 0
> write(2, "\0\0\247\0NI\0\0t2022-10-27 13:06:21.879"..., 176) = 176
> 
> I tried to google for this issue, and search the mailing list, but to
> no avail. Any tips on how to fix this would be greatly appreciated!

Pgpool-II uses SysV semaphore.

> semtimedop(1, [{7, -1, SEM_UNDO}], 1, NULL) = -1 EINVAL (Invalid argument)

This indicates that attempt to try to aquire lock using semaphore #7
but failed. At this moment the cause of the error is not clear. Can
you share output of "ipcs -s"?

Below is an output on my Ubuntu box.

$ LANG=C ipcs -s

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x00000000 131097     t-ishii    600        8         
0x00000000 229414     t-ishii    600        9         

semid 229414 shows Pgpool-II obtained 9 semaphores which is
expected. nsems varies depending on Pgpool-II version. For 4.3, it's
9. If nsems is 9, then semaphore #7 is valid. If nsems is lower than
8, then it fails.

Note that the semid 229414 was assigned when Pgpool-II started up. If
you run Pgpool-II under strace, you can see the number in the strace
log:

semget(IPC_PRIVATE, 9, IPC_CREAT|IPC_EXCL|0600) = 229414
semctl(229414, 0, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 1, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 2, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 3, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 4, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 5, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 6, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 7, SETVAL, 0x7ffd00000001) = 0
semctl(229414, 8, SETVAL, 0x7ffd00000001) = 0

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp