[pgpool-general: 8033] Re: PgPool-II 4.2.7: several segfaults on CentOS 8

Wed Mar 2 20:57:50 JST 2022

Here is the complete bt output:
===================================================================================================
DEV root at proxy2-dev ~# gdb /usr/bin/pgpool
/root/pgpool-coredumps/core.pgpool.26.e120553488834d47bff3c72a21a9bba1.694540.1646129274000000
GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8
Copyright (C) 2018 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html
>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/pgpool...Reading symbols from
/usr/lib/debug/usr/bin/pgpool-4.2.5-1pgdg.rhel8.x86_64.debug...done.
done.

warning: Can't open file (null) during file-backed mapping note processing
[New LWP 694540]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `pgpool: watchdog                                '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
120 movdqu (%rax), %xmm4
(gdb) bt
#0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
#1  0x00007fe9b70ac0af in _IO_vfprintf_internal (s=s at entry=0x7ffdb575f7c0,
format=format at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s
Current State = %s",
    ap=ap at entry=0x7ffdb575f9b0) at vfprintf.c:1638
#2  0x00007fe9b7164bfd in ___vsnprintf_chk (s=s at entry=0x55d3177fd6b8 "STATE
MACHINE INVOKED WITH EVENT = it has found us again", maxlen=<optimized
out>, maxlen at entry=1024,
    flags=flags at entry=1, slen=slen at entry=18446744073709551615,
format=format at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s
Current State = %s",
    args=args at entry=0x7ffdb575f9b0) at vsnprintf_chk.c:63
#3  0x000055d316dc9013 in vsnprintf (__ap=0x7ffdb575f9b0,
__fmt=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s Current State =
%s", __n=1024,
    __s=0x55d3177fd6b8 "STATE MACHINE INVOKED WITH EVENT = it has found us
again") at /usr/include/bits/stdio2.h:80
#4  pvsnprintf (buf=0x55d3177fd6b8 "STATE MACHINE INVOKED WITH EVENT = it
has found us again", len=len at entry=1024,
    fmt=fmt at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s
Current State = %s", args=args at entry=0x7ffdb575f9b0) at utils/psprintf.c:126
#5  0x000055d316e11eab in appendStringInfoVA (str=str at entry=0x7ffdb575f990,
fmt=fmt at entry=0x55d3177fd2a8 "STATE MACHINE INVOKED WITH EVENT = %s Current
State = %s",
    args=args at entry=0x7ffdb575f9b0) at stringinfo.c:138
#6  0x000055d316ddeabf in errmsg (fmt=fmt at entry=0x55d316f577d0 "STATE
MACHINE INVOKED WITH EVENT = %s Current State = %s") at
../../src/utils/error/elog.c:665
#7  0x000055d316e30c4b in watchdog_state_machine
(event=event at entry=WD_EVENT_I_AM_APPEARING_FOUND,
wdNode=wdNode at entry=0x55d3177fbcf8, pkt=pkt at entry=0x0,
    clusterCommand=clusterCommand at entry=0x0) at watchdog.c:5394
#8  0x000055d316e34a7b in cluster_service_message_processor
(pkt=0x55d317804898, wdNode=0x55d3177fbcf8) at watchdog.c:4030
#9  standard_packet_processor (wdNode=wdNode at entry=0x55d3177fbcf8,
pkt=pkt at entry=0x55d317804898) at watchdog.c:4202
#10 0x000055d316e312cd in watchdog_state_machine_nw_isolation
(clusterCommand=<optimized out>, pkt=<optimized out>, wdNode=<optimized
out>, event=<optimized out>)
    at watchdog.c:6458
#11 watchdog_state_machine (event=event at entry=WD_EVENT_PACKET_RCV,
wdNode=wdNode at entry=0x55d3177fbcf8, pkt=pkt at entry=0x55d317804898,
clusterCommand=clusterCommand at entry=0x0)
    at watchdog.c:5561
#12 0x000055d316e38514 in read_sockets (rmask=rmask at entry=0x7ffdb57604e0,
pending_fds_count=pending_fds_count at entry=1) at watchdog.c:8207
#13 0x000055d316e3b04f in watchdog_main () at watchdog.c:3500
#14 0x000055d316e3bb81 in fork_watchdog_child () at watchdog.c:1127
#15 initialize_watchdog () at watchdog.c:625
#16 0x000055d316d73088 in reaper () at main/pgpool_main.c:2428
#17 0x000055d316d79e28 in PgpoolMain (discard_status=<optimized out>,
clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:473
#18 0x000055d316d7053c in main (argc=<optimized out>, argv=0x7ffdb576d1b8)
at main/main.c:365
===================================================================================================

Thank you!

Il giorno mer 2 mar 2022 alle ore 05:05 Tatsuo Ishii <ishii at sraoss.co.jp>
ha scritto:

> Can you please share the result of gdb command "bt"?  Otherwise we
> cannot figure out which place of pgpool is related to the problem.
> --
> Tatsuo Ishii
> SRA OSS, Inc. Japan
> English: http://www.sraoss.co.jp/index_en.php
> Japanese:http://www.sraoss.co.jp
>
> > This is the coredump of the segmentation fault happened on proxy2-dev:
> >
> >
> ===================================================================================================
> > DEV root at proxy2-dev /v/l/s/coredump# gdb /usr/bin/pgpool
> >
> /root/pgpool-coredumps/core.pgpool.26.e120553488834d47bff3c72a21a9bba1.694540.1646129274000000
> > GNU gdb (GDB) Red Hat Enterprise Linux 8.2-16.el8
> > Copyright (C) 2018 Free Software Foundation, Inc.
> > License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.html
> >>
> > This is free software: you are free to change and redistribute it.
> > There is NO WARRANTY, to the extent permitted by law.
> > Type "show copying" and "show warranty" for details.
> > This GDB was configured as "x86_64-redhat-linux-gnu".
> > Type "show configuration" for configuration details.
> > For bug reporting instructions, please see:
> > <http://www.gnu.org/software/gdb/bugs/>.
> > Find the GDB manual and other documentation resources online at:
> >     <http://www.gnu.org/software/gdb/documentation/>.
> >
> > For help, type "help".
> > Type "apropos word" to search for commands related to "word"...
> > Reading symbols from /usr/bin/pgpool...Reading symbols from
> > /usr/lib/debug/usr/bin/pgpool-4.2.5-1pgdg.rhel8.x86_64.debug...done.
> > done.
> >
> > warning: Can't open file (null) during file-backed mapping note
> processing
> > [New LWP 694540]
> > [Thread debugging using libthread_db enabled]
> > Using host libthread_db library "/lib64/libthread_db.so.1".
> > Core was generated by `pgpool: watchdog                                '.
> > Program terminated with signal SIGSEGV, Segmentation fault.
> > #0  __strlen_sse2 () at ../sysdeps/x86_64/multiarch/../strlen.S:120
> > 120 movdqu (%rax), %xmm4
> >
> ===================================================================================================
> >
> > Thanks.
> >
> > Il giorno mar 1 mar 2022 alle ore 17:39 shacky <shacky83 at gmail.com> ha
> > scritto:
> >
> >> Hi,
> >> I am experiencing many segfaults on PgPool-II 4.2.7 installed on two
> >> different environments (production and test).
> >>
> >> Each environment is composed by three different PgPool-II nodes
> configured
> >> in a PgPool-II watchdog cluster, and I am experiencing segfaults on
> every 6
> >> PgPool-II nodes.
> >>
> >> The 6 PgPool-II nodes are distributed on 4 different physical servers,
> so
> >> it's very unlikely that it's due to some hardware problems.
> >>
> >> These are the snippets of the segfaults notifications:
> >>
> >>
> ===================================================================================================
> >> DEV root at proxy1-dev ~# dmesg|grep pgpool
> >> [   15.970081] pgpool[834]: segfault at 8 ip 00007fb6069f5d8a sp
> >> 00007fff848e7448 error 4 in libc-2.28.so[7fb60695c000+1bc000]
> >> [   17.672642] pgpool[1336]: segfault at 8 ip 00007fb6069f5d8a sp
> >> 00007fff848e73e8 error 4 in libc-2.28.so[7fb60695c000+1bc000]
> >>
> >> DEV root at proxy2-dev ~# dmesg|grep pgpool
> >> [ 2109.530239] pgpool[848]: segfault at 8 ip 00007f372d024d8a sp
> >> 00007ffe5126c7d8 error 4 in libc-2.28.so[7f372cf8b000+1bc000]
> >> [ 4752.698303] pgpool[22406]: segfault at 8 ip 00007fe9b70f3d8a sp
> >> 00007ffdb575f298 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
> >> [ 4761.219745] pgpool[22450]: segfault at 8 ip 00007fe9b70f3d8a sp
> >> 00007ffdb575f238 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
> >> [7089616.463274] pgpool[22568]: segfault at 8 ip 00007fe9b70f3d8a sp
> >> 00007ffdb575f208 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
> >> [7090394.285732] pgpool[661708]: segfault at 8 ip 00007fe9b70f3d8a sp
> >> 00007ffdb575f208 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
> >> [7090504.438435] pgpool[661929]: segfault at 8 ip 00007fe9b70f3d8a sp
> >> 00007ffdb575f208 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
> >> [7090528.565248] pgpool[661991]: segfault at 8 ip 00007fe9b70f3d8a sp
> >> 00007ffdb575f238 error 4 in libc-2.28.so[7fe9b705a000+1bc000]
> >>
> >> DEV root at proxy3-dev ~# dmesg|grep pgpool
> >> [ 2085.795853] pgpool[879]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6678 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7088526.747095] pgpool[20105]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d65e8 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7088528.639860] pgpool[946407]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7089324.627060] pgpool[946415]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7089384.787587] pgpool[946506]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7089580.047313] pgpool[946521]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7089591.340077] pgpool[946547]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7089597.788270] pgpool[946556]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7089614.064323] pgpool[946565]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7090492.104936] pgpool[946574]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d65e8 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >> [7090575.965820] pgpool[946743]: segfault at 8 ip 00007f9f45e57d8a sp
> >> 00007ffc605d6618 error 4 in libc-2.28.so[7f9f45dbe000+1bc000]
> >>
> >> PROD root at proxy1-prod ~# dmesg|grep pgpool
> >> [    6.631205] pgpool[814]: segfault at 8 ip 00007fe9729dbd8a sp
> >> 00007ffe7174ae88 error 4 in libc-2.28.so[7fe972942000+1bc000]
> >> [   17.054962] pgpool[1288]: segfault at 8 ip 00007fe9729dbd8a sp
> >> 00007ffe7174ae28 error 4 in libc-2.28.so[7fe972942000+1bc000]
> >> [868124.710835] pgpool[1415]: segfault at 8 ip 00007fe9729dbd8a sp
> >> 00007ffe7174ae28 error 4 in libc-2.28.so[7fe972942000+1bc000]
> >> [868127.082112] pgpool[1632535]: segfault at 8 ip 00007fe9729dbd8a sp
> >> 00007ffe7174ae28 error 4 in libc-2.28.so[7fe972942000+1bc000]
> >>
> >> PROD root at proxy2-prod ~# dmesg|grep pgpool
> >> [   16.783629] pgpool[835]: segfault at 8 ip 00007f5f87877d8a sp
> >> 00007ffe3d47eb08 error 4 in libc-2.28.so[7f5f877de000+1bc000]
> >> [   21.012660] pgpool[1381]: segfault at 8 ip 00007f5f87877d8a sp
> >> 00007ffe3d47eaa8 error 4 in libc-2.28.so[7f5f877de000+1bc000]
> >> [ 1978.102123] pgpool[1400]: segfault at 8 ip 00007f5f87877d8a sp
> >> 00007ffe3d47eaa8 error 4 in libc-2.28.so[7f5f877de000+1bc000]
> >>
> >> PROD root at proxy3-prod ~# dmesg|grep pgpool
> >> [   13.567572] pgpool[826]: segfault at 8 ip 00007f4359c84d8a sp
> >> 00007fff04ddf008 error 4 in libc-2.28.so[7f4359beb000+1bc000]
> >> [   15.559884] pgpool[1384]: segfault at 8 ip 00007f4359c84d8a sp
> >> 00007fff04ddefa8 error 4 in libc-2.28.so[7f4359beb000+1bc000]
> >> [1094174.197742] pgpool[1398]: segfault at 8 ip 00007f4359c84d8a sp
> >> 00007fff04ddef78 error 4 in libc-2.28.so[7f4359beb000+1bc000]
> >>
> >>
> ===================================================================================================
> >>
> >> The operating system is CentOS 8 (fully updated) with these PgPool-II
> >> packages installed:
> >>
> >>
> ===================================================================================================
> >> PROD root at proxy3 ~# rpm -qa|grep -i pgpool
> >> pgpool-II-pg12-4.2.7-1pgdg.rhel8.x86_64
> >> pgpool-II-release-4.2-1.noarch
> >>
> >>
> ===================================================================================================
> >>
> >> Could you help me to understand what's going on, please?
> >>
> >> Thank you very much!
> >> Bye
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20220302/35d7fe25/attachment.htm>