Re:'Check Your Backups Work' Day
by colechristensen on Wednesday February 01, 2017 @01:07AM
(#13538682)
Then we'll have a check your check your backups day where you make sure your automation isn't making sure the sky is blue.
[ Parent
| on HN ]
It's in there now. Row counts per table and checksums are written out with the backup for every table. If things don't match up, alarm in a loud and noisy way.
The point is that you could have a bug in the verification process that returns "all good" when it really isn't.
I have a SELECT of the most recent updated_at fields for a few tables emailed to me so I can eyeball the values
Thats a neat idea, I might look at that, thank you.
My point is that there are sanity checks built into the process. It's not difficult. There doesn't need to be a human verifying things there. Is the row count increasing over the previous backup? Check. Is the checksum different? Check. Is it non-zero? Check. And on and on.
Stop making excuses and automate testing your backups.
so, when your check process does not report because it failed to run, how do you know? Do you have a monitor for your monitor? Does that monitor have a monitor?
There is no reason not to automate your backup tests, but there is no reason not to eyeball the check is actually working from time to time.
Yes, the restore process checks in daily; alarms are thrown if it doesn't check in.
MySQL is also pretty good about not starting if the data is corrupted for whatever reason. The process not starting is a pretty obvious alarm, too.
Checks are cheap. Better to automate your logic and invest in good monitoring and automation.