True, but software for domains where people can die is very different than Cloudflare. The scale is much higher in the latter, and it needs to be less "bulletproof".

If you're writing software for giant metal flying cylinder carrying hundreds of people, we can't just brush mistakes off the same way we might with web-based software (and yes, privacy is important, but it's not life or death).

I'm not "brushing mistakes off", at all. The process that led to the bugs needs a full review and root-cause corrective action, especially enhanced testing scenarios. I'm just pointing out that software "being wrong under load" is not some sort of crazy, unusual problem that points to poor software architecture. These bugs can easily be caused by a single line of code that wasn't synchronized correctly.

... ... that's the problem. why isn't it running isolated?

In most industries that are not IT, testing the IT systems very well is needed, but rarely achieved. In my area (FDA regulated) performance testing is almost never done correctly and (for sure) never completely, but as long as it works and the auditors never look at these details nobody from management is willing to know about it, because fixing it costs money. For many years the people signing off for all the software were intentionally not business people with zero IT knowledge, so they could not be accused of knowingly signing off low quality code. That saved tens or hundreds of millions of dollars in the past 20 years alone.

> In most industries, testing the IT systems very well is needed, but rarely achieved.

FTFY. IT shops on the whole aren’t any better here.

"huge" is not the same as "horrible". One racy boi can take out an airplane.

But calling something an "internal architectural problem" implies things are fundamentally designed wrong, and the code needs a rewrite. My point was the architecture can definitely be fine, but you can still have bugs (even catastrophic ones) because someone left out a `synchronized` keyword somewhere.

It could still be

For example, one way to use SQL is to escape your parameters and then concat them straight into your SQL string. However, if someone leaves out an ‘escape’ somewhere, you have a major security incident

The other way is to pass parameters separately as data to your SQL driver. It completely negates the problem

If you’ve chosen the first way in your project already, you’ve committed to a major internal architectural problem. In the same vein, maybe if your code requires sprinkling ‘synchronized’ everywhere, you did it wrong

But that said, we don’t know what actually is the problem since it’s just a marketing statement saying that high load was the cause and that doesn’t tell us anything

That's still an architecture defect.

Software's accuracy being affected by load is completely unacceptable.

Please let us know where these software magicians are that write perfect, bug-free code.

Who said anything about that? You're arguing against a straw man nobody has set up but yourself..

There's no such thing as perfect software, but there is definitely software that doesn't make up a false value if it can't work.

And if there is a bug in the logic that determines if it can or can’t work?

fail safe. Don't fail stupid.

Hence the crumple thingy on the tail of the aircraft

For what it's worth, I had the same reaction upon reading this paragraph. It seems there are HN commenters who understand software design and those who do not.

To be fair to the other commenters, it's conceivable that the practical difference between the current unsound architecture and a sound replacement might come down to to some small defect. But an attempt to fix the problem by fixing just the defect, even some kind of RCA process on it, will fail if the architecture itself is not fixed, and that requires a change in attitudes and understanding by the devs responsible for the system, not just a software change. That is where the real flaw lies. But some people don't believe that it's possible for the flaw to be in these places, outside the code. All the RCAs in the world can't help them.