That way, you can migrate users one by one (sending a 'redirectUserConnections(userId)' to the original server, and even pick a shard that is geographically close to the user. (If you care about race conditions during migrations, you need to be really careful though.)
race conditions during migrations are the biggest concern.... You do not want a client with a connection to server they will never receive a message on.
Can't you not involve the client and send it an explicit "reconnect to server XYZ" message, only move clients that really need to move?
haven't committed to using one particular technology. queues that support retries could be useful in the event that a disconnect happens messages wouldn't be necessarily lost. Something custom built on top of Redis could accomplish the same perhaps.
You're welcome to checkout the docs and get started: https://docs.hathora.dev/#/buildkit/README
If you'd like to get in touch, feel free to shoot me a message: sid [at] hathora.dev
Then you can use whatever logic you want for it. If the client disconnects, it just asks again and reconnects.
Use a token from the directory endpoint to authorize the connection so they have to do it.
I want the connections to be distributed to separate instances of a WS server based on modulo-based or consistent hashing algorithm... above that in the stack we have a message broker or queue partitioned using the same sharding algorithm for the number of websocket servers running. This queue/message broker sends publishes messages to each web socket service which in-turn pushes the message to the client. This is how I want to handle horizontally scalable web socket connections...
The question I have is the best way to do this and the best way to migrate connections. From one service to another when/if the replication factor of web socket servers changes (we add/remove servers)...
One way I would think would be to send a disconnect message to the websocket service when this event happens and it would disconnect it's clients and then have a random delay for when re-connection happens... so as to avoid trying to reinitialize all of those connections at once. This seems sub-optimal as with consistent-hashing many of the connections will stay on the same server, and terminating those connections would be unnecessary.
What is the best way to migrate an open TCP from one server to another? Or if the connection must be closed... how can we minimize the number of connections that are to be closed.
BTW: I know I can broadcasts to all instances of the websocket server, and then unless that server holds the connection it is no-op... if the server does have a connection then send the message back... This also seems it would have a limit to how far this scaling-strategy would grow before running into limitations.