<div dir="ltr"><div dir="ltr"><div style="font-family:"comic sans ms",sans-serif">Dear Tatsuo,</div><div style="font-family:"comic sans ms",sans-serif"><br></div><div style="font-family:"comic sans ms",sans-serif">Thank you for your guidance.</div><div style="font-family:"comic sans ms",sans-serif"><br></div><div style="font-family:"comic sans ms",sans-serif">I would appreciate it if you could provide me with the specific steps on how to attach gdb to pgpool and set the breakpoint at pool_semaphore_create(). I am not familiar with this process and would value your expertise.</div><div style="font-family:"comic sans ms",sans-serif"><br></div><div style="font-family:"comic sans ms",sans-serif">Best Regards,</div><div style="font-family:"comic sans ms",sans-serif">Maria </div><div style="font-family:"comic sans ms",sans-serif"><br></div><div class="gmail_default" style="font-family:"comic sans ms",sans-serif"><br></div><input name="virtru-metadata" type="hidden" value="{"email-policy":{"disableCopyPaste":false,"disablePrint":false,"disableForwarding":false,"enableNoauth":false,"expandedWatermarking":false,"expires":false,"sms":false,"expirationNum":1,"expirationUnit":"days","isManaged":false,"persistentProtection":false},"attachments":{},"compose-id":"1","compose-window":{"secure":false}}"></div><br><div class="gmail_quote gmail_quote_container" style=""><div dir="ltr" class="gmail_attr">On Fri, Feb 14, 2025 at 7:12 AM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org">ishii@postgresql.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> Dear Tatsuo,<br>
> <br>
> Thank you for your continued support.<br>
> <br>
> I have attached the trace file and the pgpool .log for your review. I don't<br>
> see the semaphore error within the log itself.<br>
<br>
Altough the strace file does not include semget call, it seems the<br>
semaphore was actually created thus sr_check process did not emit the<br>
semaphore error. You can make sure that by using ipcs -s command.<br>
<br>
> Could you please clarify how<br>
> to identify if pgpool failed to acquire the semaphore at startup?<br>
<br>
You should see FATAL message in the log something like:<br>
<br>
Unable to create semaphores:8<br>
<br>
> Additionally, could you provide guidance on how to resolve this issue if it<br>
> has occurred?<br>
<br>
I recommend to try restarting pgpool. Please make sure that before<br>
restart pgpool. shutdown pgpool and make sure there's no pgpool<br>
related process remain.<br>
<br>
BTW, this kind of problem is really hard to resolve because I failed<br>
to reproduce the problem. If I were you, I would attach gdb to pgpool<br>
and set break point at pool_semaphore_create(), then use run command<br>
of gdb to see if pgpool actually creates semaphore or not.<br>
<br>
Best reagards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS K.K.<br>
English: <a href="https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-bbb96eab86ee7dab&q=1&e=4781524a-3201-4aca-a22a-c8a94d91ec31&u=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en%2F" rel="noreferrer" target="_blank">https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-bbb96eab86ee7dab&q=1&e=4781524a-3201-4aca-a22a-c8a94d91ec31&u=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en%2F</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
<br>
<br>
> Best Regards,<br>
> Maria<br>
> <br>
> <br>
> On Wed, Feb 12, 2025 at 9:20 PM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>> wrote:<br>
> <br>
>> [Cc:ed to pgpoo-general]<br>
>><br>
>> Hello Maria,<br>
>><br>
>> Please Cc: to pgpool-general so that our community can share valuable<br>
>> info.<br>
>><br>
>> > Hello Tatsuo,<br>
>> ><br>
>> > This is how we installed pgpool.<br>
>> ><br>
>> > mkdir -p /local/opt/points/pgpool/4.4.6<br>
>> > cd /usr/src<br>
>> > wget<br>
>> <a href="https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-6acd631f770dce77&q=1&e=50e5f7b9-15c6-4965-86ea-096504160ff4&u=https%3A%2F%2Fwww.pgpool.net%2Fmediawiki%2Fimages%2Fpgpool-II-4.4.6.tar.gz" rel="noreferrer" target="_blank">https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-6acd631f770dce77&q=1&e=50e5f7b9-15c6-4965-86ea-096504160ff4&u=https%3A%2F%2Fwww.pgpool.net%2Fmediawiki%2Fimages%2Fpgpool-II-4.4.6.tar.gz</a><br>
>> > tar -xvzf pgpool-II-4.4.6.tar.gz<br>
>> > cd pgpool-II-4.4.6/<br>
>> > ./configure --prefix=/local/opt/points/pgpool/4.4.6<br>
>> --enable-sequence-lock<br>
>> > --with-openssl<br>
>> > make ; make install<br>
>> ><br>
>> > Are we missing something?<br>
>><br>
>> No, except --enable-sequence-lock is not necessary if you run pgpool<br>
>> with streaming replication mode.<br>
>><br>
>> BTW, I suspect the source of the problem is, pgpool failed to acquire<br>
>> semaphore at the startup for some reasons. To confirm the theory (and<br>
>> possibly to know the cause), can you run pgpool with strace and share<br>
>> the output? For example:<br>
>><br>
>> strace -f --trace=ipc pgpool [options....]<br>
>><br>
>> If you succeed, you will see something like:<br>
>><br>
>> semget(IPC_PRIVATE, 8, IPC_CREAT|IPC_EXCL|0600) = 655401<br>
>> semctl(655401, 0, SETVAL, 0x7ffe00000001) = 0<br>
>> semctl(655401, 1, SETVAL, 0x7ffe00000001) = 0<br>
>> semctl(655401, 2, SETVAL, 0x7ffe00000001) = 0<br>
>> semctl(655401, 3, SETVAL, 0x7ffe00000001) = 0<br>
>> semctl(655401, 4, SETVAL, 0x7ffe00000001) = 0<br>
>> semctl(655401, 5, SETVAL, 0x7ffe00000001) = 0<br>
>> semctl(655401, 6, SETVAL, 0x7ffe00000001) = 0<br>
>> semctl(655401, 7, SETVAL, 0x7ffe00000001) = 0<br>
>><br>
>> > On Wed, Feb 12, 2025 at 10:23 AM Maria Nieva <<a href="mailto:maria.nieva@points.com" target="_blank">maria.nieva@points.com</a>><br>
>> wrote:<br>
>> ><br>
>> >><br>
>> >> Dear Tatsuo,<br>
>> >><br>
>> >> Thank you for your response.<br>
>> >><br>
>> >> I've attached the pgpool log from our production environment during the<br>
>> >> upgrade. The semaphore warning shows right away. We have since rolled<br>
>> >> back to version 4.1.1. I've also included the config file for your<br>
>> >> reference.<br>
>> >><br>
>> >> Please let me know if you have any further questions.<br>
>> >><br>
>> >><br>
>> >> Best Regards,<br>
>> >> Maria<br>
>> >><br>
>> >><br>
>> >><br>
>> >><br>
>> >> On Tue, Feb 11, 2025 at 7:40 PM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> wrote:<br>
>> >><br>
>> >>> [Cc:ed to pgpool-general]<br>
>> >>><br>
>> >>> Thank you for the log. There's no semaphore errors. Now I need to know<br>
>> >>> how you installed 4.4.6. If from RPM, did you use Pgpool-II community<br>
>> >>> version?<br>
>> >>><br>
>> >>><br>
>> <a href="https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-ba8a251b4f567a67&q=1&e=e45aec18-0dee-41a2-bde7-d39d532f71f2&u=https%3A%2F%2Fpgpool.net%2Fmediawiki%2Findex.php%2FYum_Repository" rel="noreferrer" target="_blank">https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-ba8a251b4f567a67&q=1&e=e45aec18-0dee-41a2-bde7-d39d532f71f2&u=https%3A%2F%2Fpgpool.net%2Fmediawiki%2Findex.php%2FYum_Repository</a><br>
>> >>><br>
>> >>> Can you share pgpool.conf? (please mask sensitive information such as<br>
>> >>> password or IP).<br>
>> >>><br>
>> >>> Best reagards,<br>
>> >>> --<br>
>> >>> Tatsuo Ishii<br>
>> >>> SRA OSS K.K.<br>
>> >>> English:<br>
>> >>><br>
>> <a href="https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-bbb96eab86ee7dab&q=1&e=e45aec18-0dee-41a2-bde7-d39d532f71f2&u=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en%2F" rel="noreferrer" target="_blank">https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-bbb96eab86ee7dab&q=1&e=e45aec18-0dee-41a2-bde7-d39d532f71f2&u=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en%2F</a><br>
>> >>> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>> >>><br>
>> >>><br>
>> >>> > Dear Tatsuo,<br>
>> >>> ><br>
>> >>> > Thank you for your prompt response. I have attached our pgpool log<br>
>> for<br>
>> >>> your<br>
>> >>> > review.<br>
>> >>> ><br>
>> >>> > Please advise on how to resolve the semaphore error. Your guidance is<br>
>> >>> > greatly appreciated.<br>
>> >>> ><br>
>> >>> ><br>
>> >>> ><br>
>> >>> ><br>
>> >>> > On Mon, Feb 10, 2025 at 7:09 PM Tatsuo Ishii <<a href="mailto:ishii@postgresql.org" target="_blank">ishii@postgresql.org</a>><br>
>> >>> wrote:<br>
>> >>> ><br>
>> >>> >> > Hello,<br>
>> >>> >> ><br>
>> >>> >> > We recently upgraded pgpool from 4.1.1 to 4.4.6. We noticed that<br>
>> we<br>
>> >>> are<br>
>> >>> >> > getting this<br>
>> >>> >> ><br>
>> >>> >> > 2025-02-08 01:39:33.724 PROCESS: WORKER DB: [No Connection] USER:<br>
>> [No<br>
>> >>> >> > Connection] APPL: sr_check_worker % WARNING: failed to lock<br>
>> >>> semaphore<br>
>> >>> >> [snip]<br>
>> >>> >> > Connection] APPL: sr_check_worker % LOCATION: pool_sema.c:202<br>
>> >>> >> ><br>
>> >>> >> > This doesn't show much<br>
>> >>> >> ><br>
>> >>> >> > ipcs -s<br>
>> >>> >> ><br>
>> >>> >> > ------ Semaphore Arrays --------<br>
>> >>> >> > key semid owner perms nsems<br>
>> >>> >><br>
>> >>> >> Strange. In your case you should see something like:<br>
>> >>> >><br>
>> >>> >> ------ Semaphore Arrays --------<br>
>> >>> >> key semid owner perms nsems<br>
>> >>> >> 0x00000000 655386 t-ishii 600 8<br>
>> >>> >><br>
>> >>> >> ("t-ishii" is the user name of the user who started pgpool)<br>
>> >>> >><br>
>> >>> >> > This is in our *pgpool.conf *<br>
>> >>> >> > backend_clustering_mode = 'streaming_replication'<br>
>> >>> >> > num_init_children = 250<br>
>> >>> >> > min_spare_children = 25<br>
>> >>> >> > max_spare_children = 50<br>
>> >>> >> > max_pool = 2<br>
>> >>> >> > child_life_time = 300<br>
>> >>> >> > child_max_connections = 2<br>
>> >>> >> > connection_life_time = 300<br>
>> >>> >> > client_idle_limit = 300<br>
>> >>> >> ><br>
>> >>> >> > Can you please help us understand why we see the warning? What is<br>
>> it?<br>
>> >>> >><br>
>> >>> >> Pgpool worker process complains that it cannot aquire a semaphore.<br>
>> >>> >><br>
>> >>> >> > How<br>
>> >>> >> > do we resolve this?<br>
>> >>> >><br>
>> >>> >> To resolve the problem, we need the pgpool log at the start up<br>
>> because<br>
>> >>> >> the semaphore is configured at that time.<br>
>> >>> >><br>
>> >>> >> Best reagards,<br>
>> >>> >> --<br>
>> >>> >> Tatsuo Ishii<br>
>> >>> >> SRA OSS K.K.<br>
>> >>> >> English:<br>
>> >>> >><br>
>> >>><br>
>> <a href="https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-bbb96eab86ee7dab&q=1&e=beae9c64-ec11-4d5c-8783-026e28457a81&u=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en%2F" rel="noreferrer" target="_blank">https://protect2.fireeye.com/v1/url?k=31323334-501d2dca-3131568e-454455534531-bbb96eab86ee7dab&q=1&e=beae9c64-ec11-4d5c-8783-026e28457a81&u=http%3A%2F%2Fwww.sraoss.co.jp%2Findex_en%2F</a><br>
>> >>> >> Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
>> >>> >><br>
>> >>><br>
>> >><br>
>><br>
</blockquote></div></div>