Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by disgruntledphd2 on Saturday November 04, 2023 @05:44AM
(#38139462)
Dunno about that, I've read similar internal postmortems at the FAANG I worked at.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by daxfohl on Saturday November 04, 2023 @03:13PM
(#38144178)
Everywhere I've worked requires a DR drill per service, but I've never seen anything where the whole company shuts down a DC at once across all services.
But probably we should. It's an immensely larger coordination problem, but frankly, it's probably the more common failure mode.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by disgruntledphd2 on Sunday November 05, 2023 @06:08AM
(#38150074)
The FAANG I worked at did this back in 2016-18, so that what happened to CloudFlare didn't happen to them.
Partially true in this case; I can't speak to modern CF (or won't, moreso) but a large amount of internal services were built around SQL db's, and weren't built with any sense of eventual consistency. Usage of read replicas was basically unheard of. Knowing that, and that this was normal, it's a cultural issue rather than an "oops" issue.
Flipping the whole DC data sources is a sign of what I'm describing; FAANG would instead be running services in multiple DC's rather than relying on primary/secondary architecture.