Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by marcinzm on Saturday November 04, 2023 @08:29AM
(#38140422)
> Complete and utter management failure. And customers apparently are sold what Cloudflare internally considers to be alpha quality software?
This has been my experience with AWS and GCP as well. Assume anything that's under 3 years old is not really GA quality no matter what they say publicly.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by arrakeenrevived on Saturday November 04, 2023 @12:35PM
(#38142596)
I've been involved with some new service launches at AWS, and it's a strict requirement that everything goes through some rigorous operational and security reviews that cover exactly these issues before the service can be launched as GA. Feature-wise people might consider them "alpha", but when it comes to the resilience and security of the launched features, they are held to much higher standards than what is being described in this post-mortem.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by phan on Saturday November 04, 2023 @02:43PM
(#38143933)
Your operational reviews must be lacking at AWS then (surprise surprise) then because there are so many instances where something will be released in alpha yet the documentation will still be outdated, stale and incorrect LOL.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by arrakeenrevived on Saturday November 04, 2023 @03:04PM
(#38144098)
I think you misunderstand what's being talked about in this thread. "Operations" in this context has nothing to do with external-facing documentation, and instead refers to the resilience of the service and ensuring it doesn't for example, stop working when a single data center experiences a power outage.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by marcinzm on Saturday November 04, 2023 @03:26PM
(#38144282)
"It stopped working because you did XYZ which you shouldn't have done despite it not being documented as something you shouldn't do" isn't different to a customer than a data center going down. For example, I'm sure the EKS UI was really resilient which meant little when random nodes dropped from a cluster due to the utter crap code in the official CNI network driver. My point wasn't that every cloud provider released alpha level software by the same definition but that by a customer's definition they all released alpha level software and label it GA.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by sofixa on Monday November 06, 2023 @01:55AM
(#38159613)
> This has been my experience with AWS and GCP as well. Assume anything that's under 3 years old is not really GA quality no matter what they say publicly.
GCP run multi-year betas of services and features, so I'm doubtful there were still things not ironed out for GA. Do you have some examples?
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by davewritescode on Saturday November 04, 2023 @09:05AM
(#38140684)
I’m going to leave out some details but there was a period of time where you could bypass cloudflare’s IP whitelisting by using Apple’s iCloud relay service. This was fixed but to my knowledge never disclosed.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by belter on Saturday November 04, 2023 @09:57AM
(#38141030)
There was a time when they were dumping encryption keys into search engine caches for weeks, and had the audacity to claim here, the issue was "mostly" solved. Until they were called out on it by Google Project Zero team...
"Cloudflare Reverse Proxies Are Dumping Uninitialized Memory" - https://news.ycombinator.com/item?id=13718752
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by byteknight on Saturday November 04, 2023 @09:57AM
(#38141031)
There still exist many bypasses that work in a lot of cases. There's even services for it now. Wouldn't be surprised if that or similar was a technique employed.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by sidikmat on Sunday November 05, 2023 @09:00AM
(#38151169)
Saw.t
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by brookst on Saturday November 04, 2023 @10:23AM
(#38141254)
> Complete and utter management failure
Too strong. A failure certainly, but painting this as the worst possible management failure is kind of silly.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by isbvhodnvemrwvn on Saturday November 04, 2023 @02:43PM
(#38143929)
To be honest if you take the circumstances and them spending half of their post-mortem blaming the vendor, it does look like a total shitshow.
Re:Post Mortem on Cloudflare Control Plane and Analytics Outage
by organsnyder on Saturday November 04, 2023 @12:50PM
(#38142756)
Having worked at companies with varying degrees of autonomy, in my experience a more flexible structure allows for building systems that are ultimately more resilient. Of course, there are ways to do it poorly, but that doesn’t mean it’s a “complete and utter management failure”.
Bah, who cares about such unimportant details, what's important is that ~dev velocity~ was reaaally high right until that moment!
> We were also far too lax about requiring new products and their associated databases to integrate with the high availability cluster. Cloudflare allows multiple teams to innovate quickly. As such, products often take different paths toward their initial alpha. While, over time, our practice is to migrate the backend for these services to our best practices, we did not formally require that before products were declared generally available (GA). That was a mistake as it meant that the redundancy protections we had in place worked inconsistently depending on the product.
Complete and utter management failure. And customers apparently are sold what Cloudflare internally considers to be alpha quality software?