Re:Outage post-mortem
by peterwwillis on Monday January 13, 2014 @02:21PM
(#7052686)
They completely missed the real lessons from this outage: Automation is way more fallible than human beings, and they didn't follow basic best practices to stage and test production maintenance.
Humans have a lot going for them. They can think continuously and dynamically. They can change their instructions at a whim. They can provide custom solutions immediately. And they aren't limited to one way to solve a problem.
When you have to perform a bunch of complicated changes in bulk, you might think automating it would be the best way to ensure a uniform delivery of your changes. But when a single thing is different about one environment, everything is fucked. The only way to ensure a bunch of sensitive changes go off without a hitch is to make it a manual process, even if you have to supplement it with some automated processes along the way.
In this case, Dropbox allowed their site's reliability to be dictated by the automated maintenance of production servers. It's always dangerous to make changes on a production server. But what makes it worse here is that they relied on a script to make sure everything happened perfectly, and didn't double-check the results before putting it back into production.
They didn't even back up the old data in case they needed to quickly revert, which should be a basic requirement of any production change! This isn't even disaster recovery, this is production deployment 101. How they allowed this upgrade to affect the production site is just crazy to me.
Humans have a lot going for them. They can think continuously and dynamically. They can change their instructions at a whim. They can provide custom solutions immediately. And they aren't limited to one way to solve a problem.
When you have to perform a bunch of complicated changes in bulk, you might think automating it would be the best way to ensure a uniform delivery of your changes. But when a single thing is different about one environment, everything is fucked. The only way to ensure a bunch of sensitive changes go off without a hitch is to make it a manual process, even if you have to supplement it with some automated processes along the way.
In this case, Dropbox allowed their site's reliability to be dictated by the automated maintenance of production servers. It's always dangerous to make changes on a production server. But what makes it worse here is that they relied on a script to make sure everything happened perfectly, and didn't double-check the results before putting it back into production.
They didn't even back up the old data in case they needed to quickly revert, which should be a basic requirement of any production change! This isn't even disaster recovery, this is production deployment 101. How they allowed this upgrade to affect the production site is just crazy to me.