My shock and amazement comes from the 24 hours to restart your payments system? Whaaaaat? If that is correct you guys have far, far more issues than not having tests.
If you want to be kind to your users, you stop accepting connections on a new server, wait a period of time (the longer the better), and then restart. It's a matter of how many users you're willing to disrupt for how quickly you can restart the system.
The canonical example at Justin.tv is the video system: people broadcast for days to weeks at a time. If you restart the server they're connected to, their stream will be disrupted (even if the auto-reconnect works).
We have a separate system that handles most of the complexity which is stateless, but sometimes you need to restart the actual connection-holding-daemon itself. How would you suggest doing that without disrupting service?
When someone wanted to do a deployment, we had a completely automated system that we called the cluster immune system. This would deploy the change incrementally, one machine at a time. That process would continually monitor the health of those machines, as well as the cluster as a whole, to see if the change was causing problems. If it didn't like what was going on, it would reject the change, do a fast revert, and lock deployments until someone investigated what went wrong.
- The code assumes that an external service is highly available
- The page looks glitchy in IE
- If you use that flash api, computers with an nvidia graphics card will show a green screen
- If you open and close a socket quickly, flash will hard-crash the entire browser...sometimes
- Under peak load, when a vacuum is running, this new query will hose the DB
That said, it sounds like this bug COULD have been caught by testing (though I don't know the details and can't be sure). And tests can be darn useful, when used in the right circumstance. But in the hierarchy of important things for running a production website, here's what I would list in priority order:
- Extensive monitoring and alerting
- Detailed, easily accessible logging
- Code review required and taken seriously
- Hiring good developers
- Automated deployment and rollback
- Integration testing
- Unit testing
We've been working our way down the hierarchy :-)
2) We have 100% automated deployment and rollback, and a neat internal tool called Brigade for managing it that I really should get around to open sourcing some day...