Tarsnap outage (July 2-3, 2023) postmortem

Re:Tarsnap outage (July 2-3, 2023) postmortem by Alupis on Thursday July 27, 2023 @02:09PM (#36897816)

There's an old Sys Admin saying (perhaps from Allan Jude of ScaleEngine) that goes something like "if your data doesn't exist in at least three places, it doesn't actually exist at all..."

That is to say, if Tarsnap is the only place you've keeping sensitive/important data, then you're "not doing it right" as a backup. Things happen... your hard drive can die suddenly, and a data center bursts into flames all on the same day.

[ Parent | on HN ]

Re:Tarsnap outage (July 2-3, 2023) postmortem by dryrun on Thursday July 27, 2023 @03:28PM (#36898859)

I feel like ovh will never stop earing about this. This has been, frankly, a traumatic event for many sysadmins I believe, and one that was shared by many from the same source, which is quite different from the standard variation of "that time when I erased the production database" (looking at you gitlab, but also at myself!). I mean, at this point it's between a legend and a warning tale and I don't know what else to call it. A bad Wednesday probably.

[ Parent | on HN ]

Re:Tarsnap outage (July 2-3, 2023) postmortem by vntok on Friday July 28, 2023 @07:57AM (#36905605)

> I feel like ovh will never stop earing about this.

To be fair, they deserve it a bit as they got up in flames twice .

Indeed, after the first fire, the geniuses over there collected all the UPS and batteries they could find from the DC and stored them all in a pile in a closed container... where they predictably bulged, failed, sparked and eventually triggered another fire after a couple days.

[ Parent | on HN ]