News for hackers. Stuff that matters.
Re:Building Facebook's Service Encryption Infastructure by Shish2k on Wednesday May 29, 2019 @11:28AM (#20041302)
> After several days, we finally narrowed down the issue to a bad Advanced Vector Extensions (AVX) instruction on a single CPU in our fleet

This isn't even the first time I've heard of an issue at FB being caused by a single bad CPU instruction. Working at a scale where "Problem X is a one-in-a-million edge case" and "Problem X happens several times per day" are synonymous is weird...

Re:Building Facebook's Service Encryption Infastructure by Diederich on Wednesday May 29, 2019 @11:48AM (#20041568)
At a previous organization I worked at that operated at this kind of scale, I came up with a couple of maxims:

1. If you're only 99% automated, you're dead.

2. Anything you can easily imagine going wrong is probably going wrong right now.

3. Everything else that can possibly go wrong will go wrong at some point in the not distant future.

Some of the problems we ran into were pretty fun and challenging. The longest substantial one I'm aware of took a couple of years to figure out.

Re:Building Facebook's Service Encryption Infastructure by seanyesmunt on Wednesday May 29, 2019 @01:16PM (#20042800)
Can you share the problem that took a couple of years to figure out?
Re:Building Facebook's Service Encryption Infastructure by Diederich on Wednesday May 29, 2019 @02:53PM (#20043944)
Sure. I didn't figure this out, but was relatively close to the investigation. My team provided a lot of data that ended up being used to come to root cause.

My team and extended teams managed almost 200,000 network devices, spread all over the world, most of which were Cisco, and most of which were installed in stores. And most of the switch ports were connected to customer facing Point Of Sale devices. Among these are employee facing registers and customer facing card scanners. That is, the devices you interact with whenever scan your card to pay for something.

With that many devices in that many locations in a largely unmanaged environment (the switches would be installed all over the store, often in the ceiling, and many of them experienced extreme temperatures), there were a constant stream of failures. The process to manage these failures was optimized, streamlined and largely automated.

However, it was discovered that switches were failing far more frequently in the northern Midwestern US than elsewhere, and then only in the winter.

So this wasn't a really big operational issue, but it had a substantial cost impact, and the rate was high enough that a lot of the affected stores did notice and were complaining.

Right. Very strange, very mysterious.

So, briefly, the root cause:

Apparently, people in the upper Midwest wear wool to stay warm far more frequently than other cold places, specifically the US northeast. And much of the time, the humidity is quite low. So, you have a lot of people wearing a lot of wool in low humidity air. These people generated a lot of static, which they would all too often discharge while interacting with the customer facing point of sale device. And, all too frequently, that pulse of static would end up flowing all the way back to the switch, often killing it.

I didn't follow the subsequent remediation efforts, so I don't know what if anything was done about that.

Re:Building Facebook's Service Encryption Infastructure by ianlevesque on Wednesday May 29, 2019 @04:22PM (#20044957)
Amusing that you were causing them physical pain when parting with their money at the store, like negative conditioning.
Re:Building Facebook's Service Encryption Infastructure by Diederich on Wednesday May 29, 2019 @05:58PM (#20046008)
Yes, that particular joke did surface after we discovered the problem. (:
Re:Building Facebook's Service Encryption Infastructure by laingc on Wednesday May 29, 2019 @03:55PM (#20044628)
That’s a fantastic anecdote. I would have loved to have been a fly on the wall when the results were reported to management.
Re:Building Facebook's Service Encryption Infastructure by lioeters on Wednesday May 29, 2019 @04:35PM (#20045131)
Indeed, great story. I'd love to have seen the face of the person who finally figured it out.
Re:Building Facebook's Service Encryption Infastructure by Diederich on Wednesday May 29, 2019 @04:03PM (#20044715)
Thanks.

I suspect it would not have been received as that big a deal to management or to anyone else. In those halcyon days, we were running into and usually solving all kinds of such edgy, extreme scale problems. It was a lot of work, but a hell of a lot of fun too.

Re:Building Facebook's Service Encryption Infastructure by maxpanas on Wednesday May 29, 2019 @03:52PM (#20044599)
One of the great stories right here. Thank you for sharing.
Re:Building Facebook's Service Encryption Infastructure by NetOpWibby on Wednesday May 29, 2019 @03:53PM (#20044608)
Wow, that’s one helluva edge case I’d never think of!
Re:Building Facebook's Service Encryption Infastructure by wnevets on Wednesday May 29, 2019 @05:25PM (#20045677)
but at least you get to blame the users!
Re:Building Facebook's Service Encryption Infastructure by parthdesai on Wednesday May 29, 2019 @03:05PM (#20044086)
If you are interested in something like that, i would recommend you to listen to this podcast [0]. One of the stories from what i remember is that someone had run a `delete table` command but it turned out that processor had a bit off which mapped `create table` command to `delete table`. It's really really interesting.

[0] https://softwareengineeringdaily.com/2017/06/16/google-early...

Re:Building Facebook's Service Encryption Infastructure by herpderperator on Wednesday May 29, 2019 @07:25PM (#20046787)
What does "a single bad CPU instruction" mean here? That the CPU was faulty and reliably miscalculated AVX instructions? I don't understand.
Re:Building Facebook's Service Encryption Infastructure by bdd on Wednesday May 29, 2019 @07:42PM (#20046882)
Yes. For that particular CPU, in one core (and its sibling thread (as in hyperthread)), AVX instruction is broken. It doesn't do what it is supposed to do.

Recently, again at Facebook, we found a few machines with CPUs where ADCX instruction is broken on at least one core. This is especially fatal because OpenSSL and Fizz (our TLS 1.3 impl) use these instructions for AMD64 architecture in RSA implementation.

Re:Building Facebook's Service Encryption Infastructure by sroussey on Wednesday May 29, 2019 @09:57PM (#20047685)
What kind of burn-in process is used to tease these out before deploying? Curious
Re:Building Facebook's Service Encryption Infastructure by bdd on Wednesday May 29, 2019 @10:07PM (#20047725)
Barring the specifics, we deploy new machines in batches of tens of thousands at a time and they practically don't receive any production traffic for weeks while repeatedly running burn-in tests followed by baseline daemons. If they survive, they get provisioned for production.

Infant mortality rate of CPUs--especially single socket designs--is very low compared to DIMMs and SSDs. They tend to develop these issues later in their economic lives. They are also comparatively rare to other module failures.

Re:Building Facebook's Service Encryption Infastructure by lclarkmichalek on Wednesday May 29, 2019 @04:01PM (#20044706)
It's always fun to have the space in your schedule to drill down to that level of issue. TLS is pretty useful in that way, as you can catch CPU issues as transport layer errors instead of way down in your application with strange decoding issues.