News for hackers. Stuff that matters.
Re:Facebook blames a server configuration change for yesterd... by halfnibble on Thursday March 14, 2019 @02:07PM (#19391775)
WAT. A server configuration change? What kind of server configuration can affect presumably thousands of machines replicated across the globe? I'm trying to understand this.
Re:Facebook blames a server configuration change for yesterd... by jhayward on Thursday March 14, 2019 @05:56PM (#19394227)
Most failures of this type end up being a cascading resource exhaustion problem propagated by an un- or mis-analyzed feedback or dependency path. It is frankly amazing it doesn't happen more often.

I'm excluding the other common type of long outage, the head-desking "failover didn't work, backups are horked, it'll take 10's of hours to restore/cold start" kind.

Re:Facebook blames a server configuration change for yesterd... by ceejayoz on Thursday March 14, 2019 @05:18PM (#19393850)
It's hardly unheard of.

https://en.wikipedia.org/wiki/Cascading_failure

An organization Facebook's size isn't gonna be applying configuration changes to one server at a time over SSH, either. A server configuration can easily affect thousands of machines across the globe if it's deployed to them all.

Re:Facebook blames a server configuration change for yesterd... by wyre on Thursday March 14, 2019 @05:42PM (#19394094)
Shy did I take so long for Facebook to release the cause of the outage? If they are applying configuration changes at a large level shouldnt it be fairly easy for them to figure out what was the cause?
Re:Facebook blames a server configuration change for yesterd... by ceejayoz on Thursday March 14, 2019 @05:52PM (#19394187)
That's silly. Error rates show as elevated on https://developers.facebook.com/status/dashboard/ until 11pm Pacific yesterday. The @facebook Twitter account sent out a statement basically within an hour of the start of the next business day.
Re:Facebook blames a server configuration change for yesterd... by viraptor on Thursday March 14, 2019 @11:17PM (#19396137)
Possibly because it doesn't matter to us really. The postmortem will be interesting to read if they publish it, but otherwise - it stopped working. Time to explain it to the peanut gallery is better spent dealing with the actual issue.
Re:Facebook blames a server configuration change for yesterd... by groestl on Thursday March 14, 2019 @06:08PM (#19394336)
It takes an admin to bring down a host, but it takes a configuration management system to bring down a site.
Re:Facebook blames a server configuration change for yesterd... by mikewhy on Thursday March 14, 2019 @05:24PM (#19393929)
"To err is human, but to really foul things up you need a computer."
Re:Facebook blames a server configuration change for yesterd... by sarcasmatwork on Thursday March 14, 2019 @02:09PM (#19391795)
Bad configuration in a tunnel, IP, BGP etc.

https://www.bleepingcomputer.com/news/technology/facebook-an...

Re:Facebook blames a server configuration change for yesterd... by gizmo385 on Thursday March 14, 2019 @05:52PM (#19394185)
Yes, but those are all reversible relatively quickly. They aren't something that I would think ought to take almost an entire day to resolve.
Re:Facebook blames a server configuration change for yesterd... by ceejayoz on Thursday March 14, 2019 @05:55PM (#19394213)
The side-effects of such a thing might not be as easily reversible.

I've had to sit around waiting a couple hours for a Percona database cluster to re-sync after a major networking whoops, and it only had a few hundred gigabytes of data.