News for hackers. Stuff that matters.
Re:I Broke Justin.tv by richcollins on Tuesday July 19, 2011 @02:47PM (#2782444)
Unit and Integration Tests don't always catch everything but cluster immune systems do a pretty good job on the important stuff:

When someone wanted to do a deployment, we had a completely automated system that we called the cluster immune system. This would deploy the change incrementally, one machine at a time. That process would continually monitor the health of those machines, as well as the cluster as a whole, to see if the change was causing problems. If it didn't like what was going on, it would reject the change, do a fast revert, and lock deployments until someone investigated what went wrong.

http://www.startuplessonslearned.com/2008/11/five-whys.html

Re:I Broke Justin.tv by abstractbill on Tuesday July 19, 2011 @02:57PM (#2782498)
Monitoring tends to be far more useful than tests for web apps. And despite how advanced this system sounds, it's usually far less work to monitor a web app than it is to test it.