News for hackers. Stuff that matters.
Re:Facebook, Instagram go down around the world in an appare... by rwultsch on Wednesday March 13, 2019 @02:56PM (#19382211)
Short duration: network, bad software deploy Long duration: db. If you break data, it takes a while to unbreak.

Source: Me. My career has been spent managing db's for internet scale sites.

Re:Facebook, Instagram go down around the world in an appare... by dsfyu404ed on Wednesday March 13, 2019 @03:30PM (#19382661)
I work for a smaller but comparably large platform. "If everything is down check the DB" is at the top of one of our internal monitoring websites in red.

Screw ups related to data loss are rare (I've been here years and haven't seen one with the DBs that the stuff I work with uses) but failures at this scale tend to cascade a little ways and it takes time to dig out of the hole. They probably have the problem solved but they have to spend a bunch of time synchronizing things and verifying the fix before they press the big red "go live" button.

Re:Facebook, Instagram go down around the world in an appare... by pferde on Thursday March 14, 2019 @08:06AM (#19388442)
Shouldn't the monitoring websites be able to check the DB status for you before you even look at that red text? :)
Re:Facebook, Instagram go down around the world in an appare... by dsfyu404ed on Thursday March 14, 2019 @09:51AM (#19389181)
We have a different dedicated page that gives an overviews of what's going on with the DB. The page in question is supposed to be a single stop that lets you visually get an overview of the state of the application servers and whether things are "normal" and if not allow you to quickly identify what is not normal.
Re:Facebook, Instagram go down around the world in an appare... by cheeze on Wednesday March 13, 2019 @03:07PM (#19382349)
Nothing worse than that sinking feeling of "oh fuck, we have to backfill a lot of data.
Re:Facebook, Instagram go down around the world in an appare... by WrtCdEvrydy on Wednesday March 13, 2019 @06:54PM (#19384794)
Why did my username on this site just change to 'test123'... oh, where clauses.
Re:Facebook, Instagram go down around the world in an appare... by jomkr on Thursday March 14, 2019 @03:39AM (#19387195)
Nothing worse then the page on Friday night, oh there goes my weekend.