Re:Summary of the Amazon S3 Service Disruption
by dbenhur on Friday March 03, 2017 @12:31AM
(#13780954)
> One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data.
Nope. Nope. Nope.
You test every backup by automatically restoring from it in a sandbox and verifying its integrity and functionality in the restored state.
Backups are worthless unless verified for their intended use of recovering a functioning system.
One would only actually test the backups about twice a year just to be damn sure they are still resulting in restorable data. The rest of the year it's only worth keeping an automated process reporting whether or not the things are being made, and people keeping an eye on change management to be sure no changes are made to the known-to-be working process that can break it without the new process incurring an explicit vetting cycle. Gitlab wasn't apparently testing or engaging in monitoring what was supposed to be an automated process. That's where they got burned.
Process monitoring may be boring as hell, but it's seldom wasted effort, and will prevent massive, compounded headaches from bringing operations to a chaotic halt.