I really liked Netflix's approach to testing backup scenarios by creating the Chaos Monkey[1] (Jeff Atwood had good insight as well[2].) The idea of creating a mechanism to test fail over scenarios is something that I wouldn't have thought of prior to the transparency that companies like Grasshopper have shown. So hat tip to them for opening up on a large failure; It makes the rest of the dev community smarter/better because of their honesty.

[1]: http://techblog.netflix.com/2010/12/5-lessons-weve-learned-u... [2]: http://www.codinghorror.com/blog/2011/04/working-with-the-ch...