[pgpool-general: 8041] Re: PgPool-II 4.2.7: several segfaults on CentOS 8
shacky
shacky83 at gmail.com
Tue Mar 8 00:19:47 JST 2022
Thanks Usama!
Will the patch be included in the next release?
When is it planned?
Thank you!
Il giorno gio 3 mar 2022 alle ore 11:43 Muhammad Usama <m.usama at gmail.com>
ha scritto:
> Hello,
>
> I have pushed the fix for all supported branches.
>
> Thanks
> Best Regards
> Muhammad Usama
>
>
> On Thu, Mar 3, 2022 at 8:03 AM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>
>> Hi Usama,
>>
>> Thank you!
>>
>> >> On 03-Mar-2022, at 4:51 AM, Tatsuo Ishii <ishii at sraoss.co.jp> wrote:
>> >>
>> >> Not sure at this moment because the cause of the segfault is not known
>> >> yet. Also it's not known how to reproduce the segfault. So I think
>> >> more investigation is needed.
>> >
>> > Hi,
>> > I have identified the issue and will push the fix in few hours after
>> testing. I don’t think upgrading to 4.3 will help in this case
>> >
>> > Thanks
>> >>
>> >>> Thanks!
>> >>> In the meantime, do you think that upgrading to 4.3 could solve the
>> problem?
>> >>>
>> >>> Thank you
>> >>>
>> >>>> Il giorno mer 2 mar 2022 alle ore 14:51 Tatsuo Ishii <
>> ishii at sraoss.co.jp>
>> >>>> ha scritto:
>> >>>>
>> >>>> Ok, It seems the segfault occured in watchdog.
>> >>>>
>> >>>> Usama,
>> >>>> Can you please take a look at this?
>> >>>>
>> >>>> Best reagards,
>> >>>> --
>> >>>> Tatsuo Ishii
>> >>>> SRA OSS, Inc. Japan
>> >>>> English: http://www.sraoss.co.jp/index_en.php
>> >>>> Japanese:http://www.sraoss.co.jp
>> >>>>
>> >>>> From: shacky <shacky83 at gmail.com>
>> >>>> Subject: Re: [pgpool-general: 8031] Re: PgPool-II 4.2.7: several
>> segfaults
>> >>>> on CentOS 8
>> >>>> Date: Wed, 2 Mar 2022 12:57:50 +0100
>> >>>> Message-ID: <CAPz3gm=
>> >>>> wPi_FqSJpnN99gKFBtg0Lg-0x7RiqAPdytfqra8saYQ at mail.gmail.com>
>> >>>>
>> >>>>> Here is the complete bt output:
>> >>>>>
>> >>>>
>> ===================================================================================================
>> >>>>> DEV root at proxy2-dev ~# gdb /usr/bin/pgpool
>> >>>>>
>> >>>>
>> /root/pgpool-coredumps/core.pgpool.26.e120553488834d47bff3c72a21a9bba1.694540.1646129274000000
>> >>>>> GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8
>> >>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>> >>>>> License GPLv3+: GNU GPL version 3 or later <
>> >>>> http://gnu.org/licenses/gpl.html
>> >>>>>>
>> >>>>> This is free software: you are free to change and redistribute it.
>> >>>>> There is NO WARRANTY, to the extent permitted by law.
>> >>>>> Type "show copying" and "show warranty" for details.
>> >>>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> >>>>> Type "show configuration" for configuration details.
>> >>>>> For bug reporting instructions, please see:
>> >>>>> <http://www.gnu.org/software/gdb/bugs/>.
>> >>>>> Find the GDB manual and other documentation resources online at:
>> >>>>> <http://www.gnu.org/software/gdb/documentation/>.
>> >>>>>
>> >>>>> For help, type "help".
>> >>>>> Type "apropos word" to search for commands related to "word"...
>> >>>>> Reading symbols from /usr/bin/pgpool...Reading symbols from
>> >>>>> /usr/lib/debug/usr/bin/pgpool-4.2.5-1pgdg.rhel8.x86_64.debug...done.
>> >>>>> done.
>> >>>>>
>> >>>>> warning: Can't open file (null) during file-backed mapping note
>> >>>> processing
>> >>>>> [New LWP 694540]
>> >>>>> [Thread debugging using libthread_db enabled]
>> >>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> >>>>> Core was generated by `pgpool: watchdog
>> '.
>> >>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>> >>>>> #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
>> >>>>> 120 movdqu (%rax), %xmm4
>> >>>>> (gdb) bt
>> >>>>> #0 __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
>> >>>>> #1 0x00007fe9b70ac0af in _IO_vfprintf_internal (s=s at entry
>> >>>> =0x7ffdb575f7c0,
>> >>>>> format=format at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH
>> EVENT =
>> >>>> %s
>> >>>>> Current State = %s",
>> >>>>> ap=ap at entry=0x7ffdb575f9b0) at vfprintf.c:1638
>> >>>>> #2 0x00007fe9b7164bfd in ___vsnprintf_chk (s=s at entry
>> =0x55d3177fd6b8
>> >>>> "STATE
>> >>>>> MACHINE INVOKED WITH EVENT = it has found us again",
>> maxlen=<optimized
>> >>>>> out>, maxlen at entry=1024,
>> >>>>> flags=flags at entry=1, slen=slen at entry=18446744073709551615,
>> >>>>> format=format at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH
>> EVENT =
>> >>>> %s
>> >>>>> Current State = %s",
>> >>>>> args=args at entry=0x7ffdb575f9b0) at vsnprintf_chk.c:63
>> >>>>> #3 0x000055d316dc9013 in vsnprintf (__ap=0x7ffdb575f9b0,
>> >>>>> __fmt=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s Current
>> >>>> State =
>> >>>>> %s", __n=1024,
>> >>>>> __s=0x55d3177fd6b8 "STATE MACHINE INVOKED WITH EVENT = it has
>> found
>> >>>> us
>> >>>>> again") at /usr/include/bits/stdio2.h:80
>> >>>>> #4 pvsnprintf (buf=0x55d3177fd6b8 "STATE MACHINE INVOKED WITH
>> EVENT = it
>> >>>>> has found us again", len=len at entry=1024,
>> >>>>> fmt=fmt at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT
>> = %s
>> >>>>> Current State = %s", args=args at entry=0x7ffdb575f9b0) at
>> >>>> utils/psprintf.c:126
>> >>>>> #5 0x000055d316e11eab in appendStringInfoVA (str=str at entry
>> >>>> =0x7ffdb575f990,
>> >>>>> fmt=fmt at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s
>> >>>> Current
>> >>>>> State = %s",
>> >>>>> args=args at entry=0x7ffdb575f9b0) at stringinfo.c:138
>> >>>>> #6 0x000055d316ddeabf in errmsg (fmt=fmt at entry=0x55d316f577d0
>> "STATE
>> >>>>> MACHINE INVOKED WITH EVENT = %s Current State = %s") at
>> >>>>> ../../src/utils/error/elog.c:665
>> >>>>> #7 0x000055d316e30c4b in watchdog_state_machine
>> >>>>> (event=event at entry=WD_EVENT_I_AM_APPEARING_FOUND,
>> >>>>> wdNode=wdNode at entry=0x55d3177fbcf8, pkt=pkt at entry=0x0,
>> >>>>> clusterCommand=clusterCommand at entry=0x0) at watchdog.c:5394
>> >>>>> #8 0x000055d316e34a7b in cluster_service_message_processor
>> >>>>> (pkt=0x55d317804898, wdNode=0x55d3177fbcf8) at watchdog.c:4030
>> >>>>> #9 standard_packet_processor (wdNode=wdNode at entry=0x55d3177fbcf8,
>> >>>>> pkt=pkt at entry=0x55d317804898) at watchdog.c:4202
>> >>>>> #10 0x000055d316e312cd in watchdog_state_machine_nw_isolation
>> >>>>> (clusterCommand=<optimized out>, pkt=<optimized out>,
>> wdNode=<optimized
>> >>>>> out>, event=<optimized out>)
>> >>>>> at watchdog.c:6458
>> >>>>> #11 watchdog_state_machine (event=event at entry=WD_EVENT_PACKET_RCV,
>> >>>>> wdNode=wdNode at entry=0x55d3177fbcf8, pkt=pkt at entry=0x55d317804898,
>> >>>>> clusterCommand=clusterCommand at entry=0x0)
>> >>>>> at watchdog.c:5561
>> >>>>> #12 0x000055d316e38514 in read_sockets (rmask=rmask at entry
>> >>>> =0x7ffdb57604e0,
>> >>>>> pending_fds_count=pending_fds_count at entry=1) at watchdog.c:8207
>> >>>>> #13 0x000055d316e3b04f in watchdog_main () at watchdog.c:3500
>> >>>>> #14 0x000055d316e3bb81 in fork_watchdog_child () at watchdog.c:1127
>> >>>>> #15 initialize_watchdog () at watchdog.c:625
>> >>>>> #16 0x000055d316d73088 in reaper () at main/pgpool_main.c:2428
>> >>>>> #17 0x000055d316d79e28 in PgpoolMain (discard_status=<optimized
>> out>,
>> >>>>> clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:473
>> >>>>> #18 0x000055d316d7053c in main (argc=<optimized out>,
>> >>>> argv=0x7ffdb576d1b8)
>> >>>>> at main/main.c:365
>> >>>>>
>> >>>>
>> ===================================================================================================
>> >>>>>
>> >>>>> Thank you!
>> >>>>>
>> >>>>> Il giorno mer 2 mar 2022 alle ore 05:05 Tatsuo Ishii <
>> ishii at sraoss.co.jp
>> >>>>>
>> >>>>> ha scritto:
>> >>>>>
>> >>>>>> Can you please share the result of gdb command "bt"? Otherwise we
>> >>>>>> cannot figure out which place of pgpool is related to the problem.
>> >>>>>> --
>> >>>>>> Tatsuo Ishii
>> >>>>>> SRA OSS, Inc. Japan
>> >>>>>> English: http://www.sraoss.co.jp/index_en.php
>> >>>>>> Japanese:http://www.sraoss.co.jp
>> >>>>>>
>> >>>>>>> This is the coredump of the segmentation fault happened on
>> proxy2-dev:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>
>> >>>>
>> ===================================================================================================
>> >>>>>>> DEV root at proxy2-dev /v/l/s/coredump# gdb /usr/bin/pgpool
>> >>>>>>>
>> >>>>>>
>> >>>>
>> /root/pgpool-coredumps/core.pgpool.26.e120553488834d47bff3c72a21a9bba1.694540.1646129274000000
>> >>>>>>> GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8
>> >>>>>>> Copyright (C) 2018 Free Software Foundation, Inc.
>> >>>>>>> License GPLv3+: GNU GPL version 3 or later <
>> >>>>>> http://gnu.org/licenses/gpl.html
>> >>>>>>>>
>> >>>>>>> This is free software: you are free to change and redistribute it.
>> >>>>>>> There is NO WARRANTY, to the extent permitted by law.
>> >>>>>>> Type "show copying" and "show warranty" for details.
>> >>>>>>> This GDB was configured as "x86_64-redhat-linux-gnu".
>> >>>>>>> Type "show configuration" for configuration details.
>> >>>>>>> For bug reporting instructions, please see:
>> >>>>>>> <http://www.gnu.org/software/gdb/bugs/>.
>> >>>>>>> Find the GDB manual and other documentation resources online at:
>> >>>>>>> <http://www.gnu.org/software/gdb/documentation/>.
>> >>>>>>>
>> >>>>>>> For help, type "help".
>> >>>>>>> Type "apropos word" to search for commands related to "word"...
>> >>>>>>> Reading symbols from /usr/bin/pgpool...Reading symbols from
>> >>>>>>>
>> /usr/lib/debug/usr/bin/pgpool-4.2.5-1pgdg.rhel8.x86_64.debug...done.
>> >>>>>>> done.
>> >>>>>>>
>> >>>>>>> warning: Can't open file (null) during file-backed mapping note
>> >>>>>> processing
>> >>>>>>> [New LWP 694540]
>> >>>>>>> [Thread debugging using libthread_db enabled]
>> >>>>>>> Using host libthread_db library "/lib64/libthread_db.so.1".
>> >>>>>>> Core was generated by `pgpool: watchdog
>> >>>> '.
>> >>>>>>> Program terminated with signal SIGSEGV, Segmentation fault.
>> >>>>>>> #0 __strlen_sse2 () at
>> ../sysdeps/x86_64/multiarch/../strlen.S:120
>> >>>>>>> 120 movdqu (%rax), %xmm4
>> >>>>>>>
>> >>>>>>
>> >>>>
>> ===================================================================================================
>> >>>>>>>
>> >>>>>>> Thanks.
>> >>>>>>>
>> >>>>>>> Il giorno mar 1 mar 2022 alle ore 17:39 shacky <
>> shacky83 at gmail.com>
>> >>>> ha
>> >>>>>>> scritto:
>> >>>>>>>
>> >>>>>>>> Hi,
>> >>>>>>>> I am experiencing many segfaults on PgPool-II 4.2.7 installed on
>> two
>> >>>>>>>> different environments (production and test).
>> >>>>>>>>
>> >>>>>>>> Each environment is composed by three different PgPool-II nodes
>> >>>>>> configured
>> >>>>>>>> in a PgPool-II watchdog cluster, and I am experiencing segfaults
>> on
>> >>>>>> every 6
>> >>>>>>>> PgPool-II nodes.
>> >>>>>>>>
>> >>>>>>>> The 6 PgPool-II nodes are distributed on 4 different physical
>> >>>> servers,
>> >>>>>> so
>> >>>>>>>> it's very unlikely that it's due to some hardware problems.
>> >>>>>>>>
>> >>>>>>>> These are the snippets of the segfaults notifications:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> ===================================================================================================
>> >>>>>>>> DEV root at proxy1-dev ~# dmesg|grep pgpool
>> >>>>>>>> [ 15.970081] pgpool[834]: segfault at 8 ip 00007fb6069f5d8a sp
>> >>>>>>>> 00007fff848e7448 error 4 in libc-2.28.so[7fb60695c000+1bc000]
>> >>>>>>>> [ 17.672642] pgpool[1336]: segfault at 8 ip 00007fb6069f5d8a sp
>> >>>>>>>> 00007fff848e73e8 error 4 in libc-2.28.so[7fb60695c000+1bc000]
>> >>>>>>>>
>> >>>>>>>> DEV root at proxy2-dev ~# dmesg|grep pgpool
>> >>>>>>>> [ 2109.530239] pgpool[848]: segfault at 8 ip 00007f372d024d8a sp
>> >>>>>>>> 00007ffe5126c7d8 error 4 in libc-2.28.so[7f372cf8b000+1bc000]
>> >>>>>>>> [ 4752.698303] pgpool[22406]: segfault at 8 ip 00007fe9b70f3d8a
>> sp
>> >>>>>>>> 00007ffdb575f298 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
>> >>>>>>>> [ 4761.219745] pgpool[22450]: segfault at 8 ip 00007fe9b70f3d8a
>> sp
>> >>>>>>>> 00007ffdb575f238 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
>> >>>>>>>> [7089616.463274] pgpool[22568]: segfault at 8 ip
>> 00007fe9b70f3d8a sp
>> >>>>>>>> 00007ffdb575f208 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
>> >>>>>>>> [7090394.285732] pgpool[661708]: segfault at 8 ip
>> 00007fe9b70f3d8a sp
>> >>>>>>>> 00007ffdb575f208 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
>> >>>>>>>> [7090504.438435] pgpool[661929]: segfault at 8 ip
>> 00007fe9b70f3d8a sp
>> >>>>>>>> 00007ffdb575f208 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
>> >>>>>>>> [7090528.565248] pgpool[661991]: segfault at 8 ip
>> 00007fe9b70f3d8a sp
>> >>>>>>>> 00007ffdb575f238 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
>> >>>>>>>>
>> >>>>>>>> DEV root at proxy3-dev ~# dmesg|grep pgpool
>> >>>>>>>> [ 2085.795853] pgpool[879]: segfault at 8 ip 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6678 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7088526.747095] pgpool[20105]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d65e8 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7088528.639860] pgpool[946407]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7089324.627060] pgpool[946415]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7089384.787587] pgpool[946506]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7089580.047313] pgpool[946521]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7089591.340077] pgpool[946547]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7089597.788270] pgpool[946556]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7089614.064323] pgpool[946565]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7090492.104936] pgpool[946574]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d65e8 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>> [7090575.965820] pgpool[946743]: segfault at 8 ip
>> 00007f9f45e57d8a sp
>> >>>>>>>> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
>> >>>>>>>>
>> >>>>>>>> PROD root at proxy1-prod ~# dmesg|grep pgpool
>> >>>>>>>> [ 6.631205] pgpool[814]: segfault at 8 ip 00007fe9729dbd8a sp
>> >>>>>>>> 00007ffe7174ae88 error 4 in libc-2.28.so[7fe972942000+1bc000]
>> >>>>>>>> [ 17.054962] pgpool[1288]: segfault at 8 ip 00007fe9729dbd8a sp
>> >>>>>>>> 00007ffe7174ae28 error 4 in libc-2.28.so[7fe972942000+1bc000]
>> >>>>>>>> [868124.710835] pgpool[1415]: segfault at 8 ip 00007fe9729dbd8a
>> sp
>> >>>>>>>> 00007ffe7174ae28 error 4 in libc-2.28.so[7fe972942000+1bc000]
>> >>>>>>>> [868127.082112] pgpool[1632535]: segfault at 8 ip
>> 00007fe9729dbd8a sp
>> >>>>>>>> 00007ffe7174ae28 error 4 in libc-2.28.so[7fe972942000+1bc000]
>> >>>>>>>>
>> >>>>>>>> PROD root at proxy2-prod ~# dmesg|grep pgpool
>> >>>>>>>> [ 16.783629] pgpool[835]: segfault at 8 ip 00007f5f87877d8a sp
>> >>>>>>>> 00007ffe3d47eb08 error 4 in libc-2.28.so[7f5f877de000+1bc000]
>> >>>>>>>> [ 21.012660] pgpool[1381]: segfault at 8 ip 00007f5f87877d8a sp
>> >>>>>>>> 00007ffe3d47eaa8 error 4 in libc-2.28.so[7f5f877de000+1bc000]
>> >>>>>>>> [ 1978.102123] pgpool[1400]: segfault at 8 ip 00007f5f87877d8a sp
>> >>>>>>>> 00007ffe3d47eaa8 error 4 in libc-2.28.so[7f5f877de000+1bc000]
>> >>>>>>>>
>> >>>>>>>> PROD root at proxy3-prod ~# dmesg|grep pgpool
>> >>>>>>>> [ 13.567572] pgpool[826]: segfault at 8 ip 00007f4359c84d8a sp
>> >>>>>>>> 00007fff04ddf008 error 4 in libc-2.28.so[7f4359beb000+1bc000]
>> >>>>>>>> [ 15.559884] pgpool[1384]: segfault at 8 ip 00007f4359c84d8a sp
>> >>>>>>>> 00007fff04ddefa8 error 4 in libc-2.28.so[7f4359beb000+1bc000]
>> >>>>>>>> [1094174.197742] pgpool[1398]: segfault at 8 ip 00007f4359c84d8a
>> sp
>> >>>>>>>> 00007fff04ddef78 error 4 in libc-2.28.so[7f4359beb000+1bc000]
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> ===================================================================================================
>> >>>>>>>>
>> >>>>>>>> The operating system is CentOS 8 (fully updated) with these
>> PgPool-II
>> >>>>>>>> packages installed:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> ===================================================================================================
>> >>>>>>>> PROD root at proxy3 ~# rpm -qa|grep -i pgpool
>> >>>>>>>> pgpool-II-pg12-4.2.7-1pgdg.rhel8.x86_64
>> >>>>>>>> pgpool-II-release-4.2-1.noarch
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>
>> >>>>
>> ===================================================================================================
>> >>>>>>>>
>> >>>>>>>> Could you help me to understand what's going on, please?
>> >>>>>>>>
>> >>>>>>>> Thank you very much!
>> >>>>>>>> Bye
>> >>>>>>>>
>> >>>>>>
>> >>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20220307/7a9e5229/attachment.htm>
More information about the pgpool-general
mailing list