News for hackers. Stuff that matters.
Re:Why prod is down by gozur88 on Friday November 17, 2017 @12:28PM (#15723981)
>Worst of all, our database is not backed up, so we lost our data backed up, but the backups don't work, so we lost our data sharded, so we lost our data twice NoSQL, so we're webscale, and we've been losing data for years

Heh. What I keep running into at various companies: "Backed up, but for some reason the restore is going to take three days (if it works), and it will be faster to tap original sources.

Why are we doing backups, then? Company policy.

Re:Why prod is down by Sgoettschkes on Friday November 17, 2017 @04:43PM (#15726230)
No one ever got fired for doing backups. It's "industry standard" so if you don't do it, you'll be out of your job faster than the server which went down. Backups being actually restorable in a reasonable amount of time isn't "industry standard", so nobody cares.
Re:Why prod is down by itsjustjoe on Friday November 17, 2017 @01:04PM (#15724348)
Because backing up stuff is super easy, but no one actually tests restoring data until they need to do it live.
Re:Why prod is down by apple4ever on Friday November 17, 2017 @01:11PM (#15724423)
We'll do it live! We'll write it and we'll do it live!
Re:Why prod is down by gozur88 on Saturday November 18, 2017 @12:25PM (#15729838)
It adds to the excitement.