But I distinctly remember talking about backups and how a raw sync of the innodb files wasn't really what we needed (it could sort of work in the original deploy of ma.gnolia, but would not scale as the site scaled). I'm kind of surprised that the community has been so tolerant of this failure. Personally, I'd be furious if I found out that a site I trusted lost all my data because they hadn't even tested their backup system.
Exactly - first rule of backups - test them. For a database, it can be as simple as using your nightly backup to create your dev or test database or whatever. I guess it isn't feasible with a half terrabyte database every day, but to not have at least a several week old backup set is somewhat unforgivable.
On the other hand, I bet he will never ever make the same mistake again, and it will make a lot of people on here think long and hard about their own backup strategies ...
Daily automated DB Backup. Daily automated DB restore to a seperate server. (Which also can kinda double as a hot spare as required.)
My best guess is the link screenshots were being stored in the DB.
OK so it won't handle your databases properly but that's not the point; 500G is not a troublesome amount of data these days.
For example, I am building an application on top of EC2 (IaaS) and utilizing the nice EBS system with snapshots to S3.
This is still a do-it-yourself style in many ways, you still need to understand the database system, read those docs and you still need to test restores etc. It's not all that different from the Ma.gnolia situation. The main difference is that you get some nice properties from EBS and S3 redundancy.
But even with those nice properties, like hell am I going to hang the crux of my business on S3 snapshots only. Besides technical failures (there have been S3 data inconsistencies reported), what if they cancel our account? Etc. etc.
Another thing people label "cloud computing" is something like Google's appengine (PaaS). Here, you trust Google entirely to store the data safely in the "datastore" but its only integrity that they try to supply you, not backups. I have never heard of a way to get Google to take snapshots (which are important when you or an attacker delete something that shouldn't have been, etc.).
You have to manually back up off of AppEngine, as in: http://code.google.com/appengine/articles/gae_backup_and_res...
But you should do something like that anyhow is my real point.
Cloud computing is still a single point of failure because it's a "single company of failure". It may address your SPOF problems with buying and running hardware but I'm seeing people let that throw caution to the wind. Instead of "hard disks will fail, know that will need contingency plans" the situation if you choose to go the cloud computing route would perhaps be "trust the redundancy system they try and supply but be sure to have a contingency plan because there is always some possibility you will be hosed."
If you're building with something higher level than Google AppEngine, or Engine Yard, etc., the principle still applies. Even up to the consumer level, look at those people "building their bookmarks" at the higher level, they lost a lot of their hard work because they trusted a single company/system.
As a business, I don't ever want to put someone in that position where they realize they should not have trusted one system. That deteriorates any trust they had in our brand and is just shitty all around. And the only way to begin on that path is to not do that yourself.
This should be a lesson: entrust your systems administration to people that have experience in this stuff. I'm not saying you have to find an expert and pay him or her expert prices, instead, you probably have a developer or admin friend who knows what it takes to implement a basic backup strategy that will at the least keep you from losing all of your data.
I watched a little bit of the podcast and he said he was just syncing the db files... well if it's innodb, that's not going to work. It says so in the MySQL docs. You know, in the section about backups.
The lesson here could be this: if you have a great idea, get it developed and out and the door -- awesome. Now talk to someone with experience in systems planning. Don't just throw a bunch of mac minis in a cabinet, bust out an rsync script and call it a day.