News for hackers. Stuff that matters.
Re:Facebook, Instagram go down around the world in an appare... by snazz on Wednesday March 13, 2019 @05:32PM (#19384082)
I’ve seen many systems go down over the last few days worldwide. Aside from the possibility of a mega-DDoS attack (which Facebook denies), all of these organizations have fairly diverse tech stacks to my knowledge. Google’s issue (supposedly) had to do with their Blobstore API, we don’t know what happened with Facebook, and many other, smaller services have had issues as well, including three intranet services at my workplace.

This leaves me wondering what software all these places have in common. The application layers are all different, the databases are all different, the containerization and provisioning systems are different, but I imagine that all these systems rely on two things: the global Internet backbone, and maybe the Linux kernel.

Have there been major security vulnerabilities patched lately in the Linux kernel that could have had unintended consequences?

Re:Facebook, Instagram go down around the world in an appare... by str33t_punk on Wednesday March 13, 2019 @06:11PM (#19384445)
Both companies are massive and have tons of developers. It becomes almost impossible to look at the system as a whole with the amount of changes coming through. And, you get scenarios where small failures cascade through the stack reaking havok. Often times its just one config change

Its telling that one of the hottest areas of distributed systems research these days is the boring topic of configuration management. Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques. It is one of the harder problems to solve and requires massive investment in tooling, refactoring, etc.

Re:Facebook, Instagram go down around the world in an appare... by snazz on Wednesday March 13, 2019 @06:22PM (#19384526)
You’re undeniably right about not looking at Facebook or Google as one whole system, but there have also been what seems like an unprecedented number of strange little outages (see the ones mentioned by https://news.ycombinator.com/item?id=19382418) that aren’t huge companies. My workplace had some of their own today that I haven’t heard an incident report about (it’s a pretty large company and I’m not in IT).
Re:Facebook, Instagram go down around the world in an appare... by hideo on Wednesday March 13, 2019 @10:24PM (#19386033)
>>Google, Microsoft, etc are paying researchers top dollar to figure out how to prevent massive outages through novel techniques

Curious what makes you think this. Are there specific job postings in either company that are focused on this?

Re:Facebook, Instagram go down around the world in an appare... by bpye on Thursday March 14, 2019 @02:52AM (#19387005)
I work for Microsoft, I know of at least CrystalNet [1].

[1] - https://www.microsoft.com/en-us/research/blog/eliminating-ne...

Re:Facebook, Instagram go down around the world in an appare... by shafte on Wednesday March 13, 2019 @05:37PM (#19384132)
The best explanation is coincidence, I think. I have direct knowledge of two of the incidents in the past few weeks, and they have completely unrelated causes.

Sometimes you just get unlucky!

Re:Facebook, Instagram go down around the world in an appare... by smacktoward on Wednesday March 13, 2019 @09:43PM (#19385827)
"Once is happenstance. Twice is coincidence. The third time it’s enemy action."

-- Ian Fleming (in Goldfinger)

Re:Facebook, Instagram go down around the world in an appare... by z3t4 on Thursday March 14, 2019 @04:27AM (#19387383)
If it can happen, it will happen.
Re:Facebook, Instagram go down around the world in an appare... by snazz on Wednesday March 13, 2019 @05:44PM (#19384200)
That’s certainly possible. We’re probably still too early to tell, but the innate conspiracy theorist slash pattern-matching part of my brain wants to find a probable connection.
Re:Facebook, Instagram go down around the world in an appare... by taneq on Wednesday March 13, 2019 @09:58PM (#19385900)
Maybe we're in the first week of a rogue AI's hard takeoff. ;)
Re:Facebook, Instagram go down around the world in an appare... by nubslayer2 on Thursday March 14, 2019 @02:01AM (#19386831)
>all of these organizations have fairly diverse tech stacks to my knowledge

>This leaves me wondering what software all these places have in common.

dunno what systems you're talking about, but seems likely they are mostly x86 systems and maybe even mostly using Intel hardware and microcode

those systems can-be/are rooted and more, to my knowledge

Re:Facebook, Instagram go down around the world in an appare... by felix246 on Wednesday March 13, 2019 @06:52PM (#19384777)
It could also just be coincidence.
Re:Facebook, Instagram go down around the world in an appare... by madrox on Wednesday March 13, 2019 @06:29PM (#19384590)
> This leaves me wondering what software all these places have in common.

Cisco or Arista

Re:Facebook, Instagram go down around the world in an appare... by happythought on Wednesday March 13, 2019 @07:49PM (#19385165)
FANGs all use white box hardware with “merchant silicon” meaning they buy the chipsets directly from Broadcom, Mellanox, etc. and build their own devices. However, they do all have Broadcom and Mellanox in common and Cisco, juniper, and arista do too.
Re:Facebook, Instagram go down around the world in an appare... by nwlieb on Wednesday March 13, 2019 @10:05PM (#19385934)
Some FANGs definitely use Arista/Cisco. As far as I know white box hardware is mostly for top of the rack switches (as opposed to backbone infra).
Re:Facebook, Instagram go down around the world in an appare... by packetslave on Wednesday March 13, 2019 @11:00PM (#19386181)
Depends on which backbone infra :-) Take a look at the B4 and Jupiter papers from Google, for example.