On Feb 27, 2025, at 2:09 PM, Coy Hile (BLOOMBERG/ 919 3RD A) <chile1@bloomberg.net> wrote:
We run into a bit of a strange edge case when our DBAs have to perform maintenance on a database cluster. Within a minute, the master has flipped, so new connections to the load-balancer name will get the new read-write replica.
However, when this happened last month, we aw it take about 8 minutes before we stopped seeing ERROR 1290, DG is readonly (paraphrased) in radiusd.log.
The load balancer should ideally close all open connections to the old master when the old master fails. That would fix the issue.
Is there anything in particular we can tweak in the SQL config to minimize that time between reconnections?
The problem is that the server doesn't know error 1290 is a fatal error. There are literally thousands of possible MySQL errors. MySQL "helpfully" doesn't provide any way to distinguish fatal errors from temporary ones. The result is that the server assumes any unknown error isn't fatal. And therefore keeps the connection open. There are no good way to know when to close a connection after "too many" failures, so the server can't really do much here. Perhaps we could update the SQL module configuration to allow you to define a list of errors, and what to do about them. But that's likely a low priority right now. The real problem here is that the land balancer keeps connections open to the old master, even after a fail-over has occurred. Alan DeKok.