Re:I Broke Justin.tv
by emmett on Wednesday July 20, 2011 @12:11PM
(#2786218)
How do you restart a system which holds open long-lived connections without kicking everyone off?
If you want to be kind to your users, you stop accepting connections on a new server, wait a period of time (the longer the better), and then restart. It's a matter of how many users you're willing to disrupt for how quickly you can restart the system.
The canonical example at Justin.tv is the video system: people broadcast for days to weeks at a time. If you restart the server they're connected to, their stream will be disrupted (even if the auto-reconnect works).
We have a separate system that handles most of the complexity which is stateless, but sometimes you need to restart the actual connection-holding-daemon itself. How would you suggest doing that without disrupting service?
If you want to be kind to your users, you stop accepting connections on a new server, wait a period of time (the longer the better), and then restart. It's a matter of how many users you're willing to disrupt for how quickly you can restart the system.
The canonical example at Justin.tv is the video system: people broadcast for days to weeks at a time. If you restart the server they're connected to, their stream will be disrupted (even if the auto-reconnect works).
We have a separate system that handles most of the complexity which is stateless, but sometimes you need to restart the actual connection-holding-daemon itself. How would you suggest doing that without disrupting service?