[pgpool-general: 9122] Re: Another segmentation fault

[Emond Papegaaij]

I've attached a sample pgpool.conf. These get generated from a template.
Things like identifiers will be different, but all settings are identical
to the one used in the logs. As you can see, both failover_on_backend_error
and failover_on_backend_shutdown are enabled.

I've increased the logging as you requested. It might take some time to
reproduce the errors. These segfaults do not occur very frequently. We see
about one failure every few days. I'll report back when I managed to
reproduce them with the new logging settings.

Best regards,
Emond

Op za 8 jun 2024 om 04:55 schreef Tatsuo Ishii <ishii at sraoss.co.jp>:

> > Thank you for the info.  This is really weird. By the time when the
> > execution reach here, backend->slots[2] must be filled like slot[0]
> > and slot[1] since VALID_BACKEND(2) returns true which means the
> > connection to backend 2 is valid. I need more time to investigate
> > this.
>
> I am trying to determine the code path when the segfault occurred.  For
> this purpose can you share pgpool.conf? I especially interested in
> failover_on_backend_error and failover_on_backend_shutdown settings.
>
> Also can you get pgpool.log when segfaults occur with log_min_messages =
> debug1?
>
> >> For the second crash (in pool_connection_pool.c:326), all slots are
> null:
> >> (gdb) info locals
> >> i = 0
> >> freed = 0
> >> closetime = 0
> >> oldestp = 0x7f474f36e168
> >> ret = 0x0
> >> info = 0x55b511283469 <opt_sort>
> >> p = 0x7f474f36e168
> >>
> >> (gdb) p *p
> >> $7 = {
> >>   info = 0x7f47470d5c08,
> >>   slots = {0x0 <repeats 128 times>}
> >> }
> >>
> >> (gdb) p p->slots[0]
> >> $10 = (POOL_CONNECTION_POOL_SLOT *) 0x0
> >> (gdb) p p->slots[1]
> >> $11 = (POOL_CONNECTION_POOL_SLOT *) 0x0
> >> (gdb) p p->slots[2]
> >> $12 = (POOL_CONNECTION_POOL_SLOT *) 0x0
> >
> > This is strange too. How come p->slots[0-2] is 0 if VALID_BACKEND
> > returns true. I will look into this.
> >
> >> I hope this helps.
> >>
> >> Best regards,
> >> Emond
> >>
> >> Op di 4 jun 2024 om 04:56 schreef Tatsuo Ishii <ishii at sraoss.co.jp>:
> >>
> >>> > No worries. I hope you had a good trip. Last night we triggered the
> last
> >>> > crash again. Is there anything we can do to make it easier for you to
> >>> find
> >>> > the cause?
> >>>
> >>> It would be helpful if you could share some variable values in the
> >>> core file. Since I don't have the pgpool load module when you got the
> >>> core, I cannot inspect the variables using the core you provided.
> >>>
> >>> > #0  connect_backend (sp=0x55803eb0a6b8, frontend=0x55803eb08768) at
> >>> > protocol/child.c:1076
> >>> > #1  0x000055803ce3d02a in get_backend_connection
> >>> (frontend=0x55803eb08768)
> >>> > at protocol/child.c:2112
> >>> > #2  0x000055803ce38fd5 in do_child (fds=0x55803eabea90) at
> >>> > protocol/child.c:416
> >>> > #3  0x000055803cdfea4c in fork_a_child (fds=0x55803eabea90, id=13) at
> >>> > main/pgpool_main.c:863
> >>> > #4  0x000055803cdfde30 in PgpoolMain (discard_status=0 '\000',
> >>> > clear_memcache_oidmaps=0 '\000') at main/pgpool_main.c:561
> >>> > #5  0x000055803cdfb9e6 in main (argc=2, argv=0x7ffc8cdddda8) at
> >>> > main/main.c:365
> >>>
> >>> Around protocol/child.c:1076:
> >>>                                 /* set DB node id */
> >>>                                 pool_set_db_node_id(CONNECTION(backend,
> >>> i), i);
> >>>
> >>> I want to see the values in the "backend" struct. Since CONNECTION
> >>> macro is used here, you have to do something like in a gdb session.
> >>>
> >>> p *backend->slots[0]
> >>> p *backend->slots[1]
> >>> p *backend->slots[2]
> >>>
> >>> Best reagards,
> >>> --
> >>> Tatsuo Ishii
> >>> SRA OSS LLC
> >>> English: http://www.sraoss.co.jp/index_en/
> >>> Japanese:http://www.sraoss.co.jp
> >>>
> > _______________________________________________
> > pgpool-general mailing list
> > pgpool-general at pgpool.net
> > http://www.pgpool.net/mailman/listinfo/pgpool-general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20240608/4de78dc7/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pgpool.conf
Type: application/octet-stream
Size: 41671 bytes
Desc: not available
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20240608/4de78dc7/attachment.obj>