[pgpool-hackers: 4230] Re: segmentation fault error

[Tatsuo Ishii]

> Hi, Pgpool hackers,
> 
> In Pgpool-II4.2.7, process down due to segmentation fault occurs.
> The OS is RHEL8 and PostgreSQL 13.2.
> 
> When I got the backtrace from core, it was as follows.
> 
> (gdb) bt
> #0 0x0000559ff3b977f4 in function_call_walker (node=0x1000100000000, context=0x7ffcefcf1390) at utils/pool_select_walker.c:341
> #1 0x0000559ff3b9458d in raw_expression_tree_walker (node=0x559ff531aa10, walker=0x559ff3b977e0 <function_call_walker>, context=0x7ffcefcf1390) at rewrite/pool_timestamp.c:1454
> #2 0x0000559ff3b9446d in raw_expression_tree_walker (node=0x559ff53147c8, walker=0x559ff3b977e0 <function_call_walker>, context=0x7ffcefcf1390) at rewrite/pool_timestamp.c:1518
> #3 0x0000559ff3b9646c in pool_has_function_call (node=node at entry=0x559ff53147c8) at utils/pool_select_walker.c:73
> #4 0x0000559ff3b7bb6c in Execute (frontend=frontend at entry=0x559ff52dbb78, backend=backend at entry=0x7f545af010e8, len=9, contents=contents at entry=0x559ff52e1b98 "C_60") at protocol/pool_proto_modules.c:1077
> #5 0x0000559ff3b8205e in ProcessFrontendResponse (frontend=frontend at entry=0x559ff52dbb78, backend=backend at entry=0x7f545af010e8) at protocol/pool_proto_modules.c:2708
> #6 0x0000559ff3b739e3 in read_packets_and_process (frontend=<optimized out>, backend=<optimized out>, reset_request=<optimized out>, state=<optimized out>, num_fields=<optimized out>, cont=<optimized out>)
>     at protocol/pool_process_query.c:4941
> #7 0x0000559ff3b742b3 in pool_process_query (frontend=<optimized out>, backend=<optimized out>, reset_request=<optimized out>) at protocol/pool_process_query.c:291
> #8 0x0000559ff3b6ce8a in do_child (fds=fds at entry=0x559ff530de90) at protocol/child.c:449
> #9 0x0000559ff3b41527 in fork_a_child (fds=0x559ff530de90, id=731) at main/pgpool_main.c:682
> #10 0x0000559ff3b46415 in failover () at main/pgpool_main.c:1996
> #11 0x0000559ff3b4579b in sigusr1_interupt_processor () at main/pgpool_main.c:1329
> #12 0x0000559ff3b48e4a in PgpoolMain (discard_status=<optimized out>, clear_memcache_oidmaps=<optimized out>) at main/pgpool_main.c:473
> #13 0x0000559ff3b3f54c in main (argc=<optimized out>, argv=0x7ffcefd01658) at main/main.c:365
> (gdb)
> 
> The reason for the abnormal termination is that the value of the argument
> node passed to function_call_walker is invalid.

Yeah.

> #0 0x0000559ff3b977f4 in function_call_walker (node=0x1000100000000, context=0x7ffcefcf1390) at utils/pool_select_walker.c:341

This jumped to:

case T_InsertStmt:

which is absolutely nonsense because the stack #1 is processing
T_List. There's no such a thing in a parse tree which consists of list
of INSERT statements.

> After analyzing the core, the caller is raw_expression_tree_walker,
> but the value of nodeTag(node) passed here is 0xE2 and T_List.
> Here, the ListCell pointer retrieved by foreach() is invalid.
> 
> 
> List structure
> (gdb) x/8gx $rbx
> 0x559ff531aa10: 0x0000000f000000e2 0x000000000000000f
> 0x559ff531aa20: 0x0000559ff5314000 0x0000559ff531a9c0
> 0x559ff531aa30: 0x0000000000000008 0x00007f545af1a300
> 0x559ff531aa40: 0x0000006469727375 0x0000000000000020
> 
> (gdb) x/8gx 0x0000559ff5314000
> 0x559ff5314000: 0x0000000000000000 0x0000000000000000
> 0x559ff5314010: 0x0000000000000000 0x0001000100000000 <- argument of function_call_walker
> 0x559ff5314020: 0x0000000000000000 0x0000000000000000
> 0x559ff5314030: 0x0000000000000000 0x0000000000000000
> 
> The previous area is also cleared to 0.
> (gdb) x/4gx 0x0000559ff5314000-0x10
> 0x559ff5313ff0: 0x0000000000000000 0x0000000000000000
> 0x559ff5314000: 0x0000000000000000 0x0000000000000000
> 
> I think that it causes area(memory) destruction, but there is no corresponding bug fix
> in the release notes after 4.2.8.
> Is it a phenomenon that is recognized as a known defect?

No. This is new to me.

> If it is not a known defect, I would like to identify the process causing the area destruction.
> What kind of method is there?

I think reproducible test cases are required.

> Note that this has only occurred once in the production environment,  and at this time we have not been able to reproduce i

Can you share pgpool.conf and pgpool log?

Best reagards,
--
Tatsuo Ishii
SRA OSS LLC
English: http://www.sraoss.co.jp/index_en/
Japanese:http://www.sraoss.co.jp