GitLab Database Incident – Live Report

Re:GitLab Database Incident – Live Report by ChuckMcM on Tuesday January 31, 2017 @08:33PM (#13537339)

   > Did the person who set up that S3 job simply write a 
   > script or something and just go "yep, it's done" and 
   > walk away?

I don't know of course, but one failure mode that has to be explicitly tested for is continual monitoring that the existing backup process is still working. We had a backup process at Blekko which stopped working once when an S3 credential that appeared unrelated was removed, as I recall it was a Nagios test that detected that the next set of backups were too small and got that fixed.

[ Parent | on HN ]

Re:GitLab Database Incident – Live Report by problems on Tuesday January 31, 2017 @09:46PM (#13537739)

Yeah, the only solution is to test your backups - either automatically or manually through failure.

My small business uses a global network of cheap servers to provide low latency to customers, so we have encountered significant lengthy failures of disk or network once every few months, but hey, we can be damn sure our backups and deploy scripts work because we're forced to restore them all too often.

[ Parent | on HN ]