Re:Building Facebook's Service Encryption Infastructure
by sroussey on Wednesday May 29, 2019 @09:57PM
(#20047685)
What kind of burn-in process is used to tease these out before deploying? Curious
Re:Building Facebook's Service Encryption Infastructure
by bdd on Wednesday May 29, 2019 @10:07PM
(#20047725)
Barring the specifics, we deploy new machines in batches of tens of thousands at a time and they practically don't receive any production traffic for weeks while repeatedly running burn-in tests followed by baseline daemons. If they survive, they get provisioned for production.
Infant mortality rate of CPUs--especially single socket designs--is very low compared to DIMMs and SSDs. They tend to develop these issues later in their economic lives. They are also comparatively rare to other module failures.
Recently, again at Facebook, we found a few machines with CPUs where ADCX instruction is broken on at least one core. This is especially fatal because OpenSSL and Fizz (our TLS 1.3 impl) use these instructions for AMD64 architecture in RSA implementation.