However, it really isn't fair to blame this kind of problem on the end user.

Are website developers also expected to keep the servers secure? If Apache isn't patched and up to date, is that the website admin's fault?

Service providers are paid to do a job. This one failed tremendously, and should lose a large chunk of business for it. I think Atwood is being far too kind in accepting 50% of the responsibility for the data loss.

To put it another way: the only reason to maintain your own backups of your site data -- aside from healthy paranoia -- is because you expect your service provider to fail at doing their job. And if that's the case, shouldn't you be finding a service provider that does it better?

If you don't expect your service provider to fail, you don't know anything about service providers.

Everybody fails.

Our colocation facility has redundant generator systems. They're tested regularly, and have handled failures previously. Yet, when the power went out, three of the backup generators failed, and our site (as well as Craigslist, Yelp, and others) was out for 45 minutes.

The cause? A bug in the backup generator's software: http://365main.com/status_update.html

Shit happens. Sometimes it's not your fault. You still need to prepare for it.

Getting hacked isn't as potentially catastrophic as not having backups. With backups, being hacked can be recovered from and the service provider changed.

>To put it another way: the only reason to maintain your own backups of your site data -- aside from healthy paranoia -- is because you expect your service provider to fail at doing their job.

A business doesn't expect its premises to burn down, but most have fire insurance in the event that this happens. Even if I don't expect my service provider to fail, there's no way to know that they won't and it makes sense to deal with this risk if the cost of dealing with it is reasonable and the cost of not dealing with it is catastrophic.