Re:Cloudflare outage should not have happened
by dpark on Wednesday November 26, 2025 @01:31PM
(#46060794)
> as soon as they become the cause of an outage they have invalidated their whole reason for existence
This is a bar no engineering effort has ever met. “If you ever fail, even for a moment, there’s no reason for you to even exist.”
There have been 6 fatal passenger airplane crashes in the US this year alone. NASA only built 6 shuttles and 2 of those exploded, killing their crews. And these were life-preserving systems that failed.
Discussions around software engineering quality always seem to veer into spaces where we assign almost mythic properties to other engineering efforts in an attempt to paint software engineering as lazy or careless.
Re:Cloudflare outage should not have happened
by stoneforger on Thursday November 27, 2025 @12:13AM
(#46065910)
The NASA example should highlight the normalisation of deviance. The Challenger o-rings had failed before and while engineers were very vocal about that, management overruled them. The foam impacts and tile loss were also a known factor in the Columbia disaster but the abort window is very small. Both point to perverse incentives: maintaining the gravy train. One comment made the point earlier that if Cloudflare were more thorough they would not have captured the market because they would be slower. Slow is smooth and smooth is fast but YMMV. At the end of the day everything can be tracked down to a system that incentivizes wealth accumulation over capability with the fixation that capability can be bought which is a lie.
A better comparison would be to compare this to airline passengers paying for their tickets, they pay a few hundred bucks in the expectation that they will arrive at their destination.
Besides, it is not the customers that determine Cloudflare's business model, Cloudflare does. Note that their whole business is to prevent outages and that as soon as they become the cause of an outage they have invalidated their whole reason for existence. Of course you could then turn this into a statistical argument that as long as they prevent more outages than they cause that they are a net benefit but that's not what this discussion is about, it is first and foremost about the standard of development they are held up against.
Ericsson identified similar issues in their offering long ago and created a very capable solution and I'm wondering if that would not have been a better choice for this kind of project, even if it would have resulted in more resource consumption.