<div dir="ltr">Tatsuo,<br><br>As requested, I re-ran pgbench with the log_per_node_statement option turned on.<br>I included just a small excerpt, but everything I saw looked to be the same pattern.<br><br>$ pgbench -h 10.1.1.1 -p1234 -U pgbench -c 75 -j 12 -t 500 -P 15 -r pgbench1 -M extended<br>starting vacuum...end.<br>progress: 15.0 s, 1044.2 tps, lat 69.690 ms stddev 7.344<br>progress: 30.0 s, 1060.1 tps, lat 69.940 ms stddev 7.635<br>transaction type: <builtin: TPC-B (sort of)><br>scaling factor: 100<br>query mode: extended<br>number of clients: 75<br>number of threads: 12<br>number of transactions per client: 500<br>number of transactions actually processed: 37500/37500<br>latency average = 69.725 ms<br>latency stddev = 7.464 ms<br>tps = 1003.125524 (including connections establishing)<br>tps = 1004.224113 (excluding connections establishing)<br>statement latencies in milliseconds:<br> 0.001 \set aid random(1, 100000 * :scale)<br> 0.001 \set bid random(1, 1 * :scale)<br> 0.000 \set tid random(1, 10 * :scale)<br> 0.000 \set delta random(-5000, 5000)<br> 11.958 BEGIN;<br> 8.933 UPDATE pgbench_accounts SET abalance = abalance + :delta WHERE aid = :aid;<br> 8.921 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;<br> 9.230 UPDATE pgbench_tellers SET tbalance = tbalance + :delta WHERE tid = :tid;<br> 10.102 UPDATE pgbench_branches SET bbalance = bbalance + :delta WHERE bid = :bid;<br> 8.907 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);<br> 11.671 END;<br><br><br><br><br>-----------------------<br><br>Looped through the logfile to grab the connections coming from pgbench and then find those:<br><br>for i in `cat pgpool-Tue.log | grep "connecting\ host" | grep "10\.1\.1\.150"|awk '{print $4}'|sed "s/://"`;do cat pgpool-Tue.log |grep $i;done|less<br><br>A snippet from there:<br>NOTE: node_id 0 and 1 are the "local" db_nodes<br>node_id 2 is the "remote" db_node<br>node_id 1 is the current primary.<br><br><br><br>2024-01-23 10:10:03: pid 4507: LOG: new connection received<br>2024-01-23 10:10:03: pid 4507: DETAIL: connecting host=10.1.1.150 port=52802<br>2024-01-23 10:10:03: pid 4507: LOCATION: child.c:1860<br>2024-01-23 10:10:03: pid 4507: LOG: statement: select count(*) from pgbench_branches<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:224<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: SELECT pg_catalog.version()<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: SELECT count(*) FROM pg_catalog.pg_proc AS p, pg_catalog.pg_namespace AS n WHERE p.proname = 'count' AND n.oid = p.pronamespace AND n.nspname ~ '.*' AND p.provolatile = 'v'<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: select count(*) from pgbench_branches<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: vacuum pgbench_branches<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:224<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: vacuum pgbench_branches<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: vacuum pgbench_tellers<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:224<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: vacuum pgbench_tellers<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: truncate pgbench_history<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:224<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: truncate pgbench_history<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: DISCARD ALL<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:224<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: DISCARD ALL<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: DISCARD ALL<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: DISCARD ALL<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: frontend disconnection: session time: 0:00:00.215 user=pgbench database=pgbench1 host=10.1.1.150 port=52802<br>2024-01-23 10:10:03: pid 4507: LOCATION: child.c:2076<br>2024-01-23 10:10:03: pid 4507: LOG: new connection received<br>2024-01-23 10:10:03: pid 4507: DETAIL: connecting host=10.1.1.150 port=52828<br>2024-01-23 10:10:03: pid 4507: LOCATION: child.c:1860<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: Parse: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Parse: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: Parse: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: Bind: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Bind: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: Bind: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:914<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: Execute: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Execute: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: Execute: BEGIN;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Parse: UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Bind: UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:914<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Execute: UPDATE pgbench_accounts SET abalance = abalance + $1 WHERE aid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Parse: SELECT abalance FROM pgbench_accounts WHERE aid = $1;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Bind: SELECT abalance FROM pgbench_accounts WHERE aid = $1;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: SELECT abalance FROM pgbench_accounts WHERE aid = $1;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:914<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Execute: SELECT abalance FROM pgbench_accounts WHERE aid = $1;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Parse: UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Bind: UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:914<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Execute: UPDATE pgbench_tellers SET tbalance = tbalance + $1 WHERE tid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Parse: UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Bind: UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:914<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Execute: UPDATE pgbench_branches SET bbalance = bbalance + $1 WHERE bid = $2;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Parse: INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($1, $2, $3, $4, CURRENT_TIMESTAMP);<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Bind: INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($1, $2, $3, $4, CURRENT_TIMESTAMP);<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($1, $2, $3, $4, CURRENT_TIMESTAMP);<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:914<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Execute: INSERT INTO pgbench_history (tid, bid, aid, delta, mtime) VALUES ($1, $2, $3, $4, CURRENT_TIMESTAMP);<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: Parse: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Parse: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: Parse: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: Bind: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Bind: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: Bind: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: D message<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: statement: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:914<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 0 backend pid: 19865 statement: Execute: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 1 backend pid: 6857 statement: Execute: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br>2024-01-23 10:10:03: pid 4507: LOG: DB node id: 2 backend pid: 17091 statement: Execute: END;<br>2024-01-23 10:10:03: pid 4507: LOCATION: pool_proto_modules.c:3596<br><br>--<div>Brian Ismay</div><div><a href="mailto:jismay@cenic.org">jismay@cenic.org</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Sat, Jan 20, 2024 at 10:39 PM Tatsuo Ishii <<a href="mailto:ishii@sraoss.co.jp">ishii@sraoss.co.jp</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">> Any recommendations would be greatly appreciated. So far, after making<br>
> multiple tuning and settings changes there have been no meaningful changes.<br>
> <br>
> What we are highly suspicious of is that the latency numbers seem to go<br>
> from around .2 ms to around 8 ms. This is very close to the round-trip-time<br>
> between our datacenters.<br>
<br>
> statement latencies in milliseconds:<br>
> 0.003 \set aid random(1, 100000 * :scale)<br>
> 0.001 \set bid random(1, 1 * :scale)<br>
> 0.001 \set tid random(1, 10 * :scale)<br>
> 0.001 \set delta random(-5000, 5000)<br>
> 8.844 BEGIN;<br>
> 9.695 UPDATE pgbench_accounts SET abalance = abalance + :delta<br>
> WHERE aid = :aid;<br>
> 8.878 SELECT abalance FROM pgbench_accounts WHERE aid = :aid;<br>
> 685.873 UPDATE pgbench_tellers SET tbalance = tbalance + :delta<br>
> WHERE tid = :tid;<br>
> 92.049 UPDATE pgbench_branches SET bbalance = bbalance + :delta<br>
> WHERE bid = :bid;<br>
> 8.826 INSERT INTO pgbench_history (tid, bid, aid, delta, mtime)<br>
> VALUES (:tid, :bid, :aid, :delta, CURRENT_TIMESTAMP);<br>
> 8.847 END;<br>
<br>
If BEGIN and END were sent to the remote standby node, then the<br>
latency number makes sense. But theoretically it should not happen<br>
because you turn off load balance mode. In this case everything should<br>
be sent to primary except the connection establishing. But even if<br>
load balance mode is off, pgpool establish connections to both primary<br>
and standbys. However the default option of pgbench is establishing<br>
connection only once at the beginning of session, meaning the<br>
connection overhead should only happen only once in the session. Thus<br>
I can not think of plausible reason for the latency numbers.<br>
<br>
Anyway, can you turn on log_per_node_statement and see everything is<br>
sent to standby? I want to confirm that first.<br>
<br>
Best reagards,<br>
--<br>
Tatsuo Ishii<br>
SRA OSS LLC<br>
English: <a href="http://www.sraoss.co.jp/index_en/" rel="noreferrer" target="_blank">http://www.sraoss.co.jp/index_en/</a><br>
Japanese:<a href="http://www.sraoss.co.jp" rel="noreferrer" target="_blank">http://www.sraoss.co.jp</a><br>
</blockquote></div>