[pgpool-general: 8557] Re: Issues taking a node out of a cluster

[Emond Papegaaij]

On Tue, Jan 24, 2023 at 8:41 AM Tatsuo Ishii <ishii at sraoss.co.jp> wrote:

> > Under some circumstances the faulty node can cause disruption of service.
> > This is especially the case when a link between nodes is unreliable. For
> > example, we've seen problems when one of the interlinks is not working
> > (i.e. node 1 and node 2 cannot communicate with each other, but can both
> > communicate with node 0). Such a scenario can cause various different
> > failures. Temporarily taking a node out of the cluster allows for the
> > administrators to troubleshoot the issue without having to worry about
> > causing interference on the service.
>
> The problem description is too vague. Please provide concrete example
> of the problem.
>

I understand. I think the case below is much more concrete: a server is
being decommissioned to be replaced by a new server. First adding the new
server and then removing the old one.


> > Another case we've seen is where one of the nodes needs to be migrated
> to a
> > different site. If it is not possible to perform a live migration of the
> > VM, the second best alternative is to first add a new node to the cluster
> > and then remove the old one. The same steps need be performed when a vm
> > host has had a catastrophic hardware failure and a node is lost
> > permanently. Our entire software stack, with the exception of pgpool,
> > supports performing these kinds of modifications without introducing any
> > downtime.
>
> Pgpool-II consists of many child process. Each process handles a user
> session and multiple backends. There are tons of places in the child
> process something like:
>
> for (i = 0; i < number_of_backends ; i++)
> {
>         do something...
> }
>
> If the backend configuration has been changed in the middle of the
> loop, the code will produce unpredictable errors. To prevent that, we
> need to handle the loop as a critical section, which requires a
> locking. This will cause serious performance degradation due to the lock
> contention.
>

I understand, and I don't mind having to restart a single pgpool instance
to fully reload the configuration. However, in the current situation, this
causes miscommunication between the restarted node and the other nodes that
still run the old configuration. The only way to fix that, is to stop all
pgpool instances and then start them again, but that will cause a major
disruption of service.

Note that adding new backend is possible without restarting
> pgpool,i.e. by reloading the configuration file. For example if you
> have 3 backends (0, 1 and 2) and want to remove backend 1,
>
> < cut steps for adding a new backend >
> 7) optionaly you can remove backend 1 configuration
>    parameters. "status" columbn of show pool_nodes will be "unused" after
>    restarting pgpool.


We've tried leaving holes in the numbering initially, but something didn't
work out as expected. Unfortunately, I don't remember the exact problem.
Maybe it had to do with each node also running a pgpool instance and gaps
were not allowed in the watchdog config (like hostname0)? I'll try a build
without the renumbering and report back with the findings. If we can indeed
leave gaps in the backend numbering, that would probably fix the issue for
us. I'm not yet sure what to do with the watchdogs though.

Best regards,
Emond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://www.pgpool.net/pipermail/pgpool-general/attachments/20230124/d12599b8/attachment.htm>