System redundancy rarely covers software faults.

You'd hope an air traffic control centre would have a big box of popsicle sticks and black pen markers to cover the fallover of power | backup power | digital systems.

Well, they do (figuratively). That is why the entire airspace is not totally shut down, but capacity "just" substantially reduced.

I believe that's literally what they fail over to, yes.

Per aircraft physical batons were literally the way a number of pre digital air traffic control systems worked.

Whoever has the stick is responsible for the aircraft - it was a literal "unique token".

Because large systems are complex and have very complex and hard to detect failure modes.

What is more impressive is how rare these events are, given the complexity of the underlying systems. Redundancy is not without its own problems (source of truth, for instance).

They aren't that rare. Last UK ATC crash was in 2014 I think:

https://www.ft.com/content/65544730-8216-11e4-b9d0-00144feab...

And that's just the UK. Airport systems crashing is a regular occurrence. Paris crashed due to still relying on a Windows 3.1 system:

https://www.zdnet.com/article/a-23-year-old-windows-3-1-syst...

The IT is just poor. Tech firms routinely change much more massive and complex systems at a far faster pace compared to the stagnation found in airline IT, and yet total failures are not more frequent.

If it makes the news it is almost by definition rare.

Im not sure that's true. If something makes the news it's likely to be of interest to a large amount of people, that doesn't immediately qualify it as something rare.

Musk's foolish actions while owner of Twitter are not rare yet always on the news

Not to forget the failure of Frankfurt airport systems earlier this year due to the breakage of a single glass-fibre cable:

https://www.dw.com/en/lufthansa-system-failure-causes-massiv...

Though this is not due to outdated IT, but lack of redundancy of a critical component of the system instead.

A lot of times large scale outages like this are because of the redundancy. The whole system is interlinked with automatic failover. then it hits a corner case that was not engineered into the fail model and you get cascading system failure where each node starts bringing down other nodes automatically. basically the lesson is: In highly interlinked systems you get highly interlinked failures.

And then after a lot of angry words and finger pointing this new failure gets added to the failure model.

My personal takeaway after chasing the long tail of automatic failover on a few projects, is quite often it is better to drop a few 9's from your service goal, decouple some of the systems, and accept that while some parts of the system may go down, it should not bring everything down with it.

Complexity tends to hide problems. An over-engineered system is going to be less stable than a simple one where you know how things can break and how to bring them online pretty quickly.

The US air traffic control crash at the start of the year turned out to be a decades old system which had not been upgraded, I think the UK one is a little bit more recent, but like alot of software, people dont want to pay for the real costs of software developments.

They agiled software to the point where it’s written by product managers and wanna be engineers that cargo cult “good practice” without understanding what it’s meant to do. This types of issues are common in british made software.