Yes... but this seems to be then a timing issue to me... what happens if the replication factor changes again while they are reconnecting to the wrong server... then they will miss the second 'reconnect' message... and will stay connected to this server where it will never receive messages... perhaps an edge-case scenario.

I recommend storing the server topology somewhere that's super highly available, like Cloudflare. Then having the client choose from that. If their connection drops (use heartbeats), retry a different one you have and refresh the server topology in parallel. If the second connection fails, retry again with the new topology. This is convergent and avoids putting load on your database to connect, so you don't suffer from thundering herd problems.