Netflix (who also survived the outage) wrote a blogpost last year in which they talked about a system they call "Chaos Monkey" which does this exact thing.
http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u...
Re:How SmugMug survived the Amazonpocalypse
by joegester on Monday April 25, 2011 @02:29PM
(#2482325)
I don't think they meant that they were doing that on live systems. Sounds like a debugging tool.
Re:How SmugMug survived the Amazonpocalypse
by boredguy8 on Monday April 25, 2011 @11:23AM
(#2481515)
If the point is "test your supposed redundancy," I agree 100%. But please, for your customers' sake, don't do it in the middle of the day, especially if it's your first test.
Re:How SmugMug survived the Amazonpocalypse
by SoftwareMaven on Monday April 25, 2011 @12:14PM
(#2481752)
To be clear, he did say start practicing during planned maintenance times, but I agree: your first try on a live system shouldn't be during peak times.
It sounds stupid but if you really do have a resilient and redunant infrastructure it shouldn't matter. If you fear someone randomly unplugging things then you have work to do ;-)