Re:UK Air traffic control network crash
by defrost on Monday August 28, 2023 @08:43AM
(#37293111)
You'd hope an air traffic control centre would have a big box of popsicle sticks and black pen markers to cover the fallover of power | backup power | digital systems.
Re:UK Air traffic control network crash
by jacquesm on Monday August 28, 2023 @08:51AM
(#37293188)
Because large systems are complex and have very complex and hard to detect failure modes.
What is more impressive is how rare these events are, given the complexity of the underlying systems. Redundancy is not without its own problems (source of truth, for instance).
The IT is just poor. Tech firms routinely change much more massive and complex systems at a far faster pace compared to the stagnation found in airline IT, and yet total failures are not more frequent.
Re:UK Air traffic control network crash
by Dfiesl on Monday August 28, 2023 @11:34AM
(#37295760)
Im not sure that's true. If something makes the news it's likely to be of interest to a large amount of people, that doesn't immediately qualify it as something rare.
Re:UK Air traffic control network crash
by somat on Monday August 28, 2023 @09:13AM
(#37293408)
A lot of times large scale outages like this are because of the redundancy. The whole system is interlinked with automatic failover. then it hits a corner case that was not engineered into the fail model and you get cascading system failure where each node starts bringing down other nodes automatically. basically the lesson is: In highly interlinked systems you get highly interlinked failures.
And then after a lot of angry words and finger pointing this new failure gets added to the failure model.
My personal takeaway after chasing the long tail of automatic failover on a few projects, is quite often it is better to drop a few 9's from your service goal, decouple some of the systems, and accept that while some parts of the system may go down, it should not bring everything down with it.
Re:UK Air traffic control network crash
by ExoticPearTree on Monday August 28, 2023 @09:53AM
(#37293970)
Complexity tends to hide problems. An over-engineered system is going to be less stable than a simple one where you know how things can break and how to bring them online pretty quickly.
Re:UK Air traffic control network crash
by darkclouds on Monday August 28, 2023 @10:37AM
(#37294721)
The US air traffic control crash at the start of the year turned out to be a decades old system which had not been upgraded, I think the UK one is a little bit more recent, but like alot of software, people dont want to pay for the real costs of software developments.
Re:UK Air traffic control network crash
by gumballindie on Monday August 28, 2023 @08:06PM
(#37301882)
They agiled software to the point where it’s written by product managers and wanna be engineers that cargo cult “good practice” without understanding what it’s meant to do. This types of issues are common in british made software.
Re:UK Air traffic control network crash
by pdx_flyer on Monday August 28, 2023 @06:43PM
(#37301335)
Lots of US bound traffic took very southerly routings today to avoid UK airspace. My guess is that this will continue until the outage is completely resolved.
Re:UK Air traffic control network crash
by gumballindie on Monday August 28, 2023 @08:00PM
(#37301842)
I am genuinely convinced that the UK is in collapse. The nhs is flattened, policing is meh, the infrastructure is pretty banged up, bills are through the roof, inflation and interest rates are mad high, and now this. Not to mention a noticeable drop in service quality throughout the private sector, with everyone blaming “staff shortage” and a government desperate to suppress the very same wages that are meant to attract more qualified staff. Seriously this is madness.
Re:UK Air traffic control network crash
by thorin on Thursday August 31, 2023 @04:46AM
(#37334238)
Most of these are global issues though in the developed world, even healthcare to an extent due to ageing population. I'm not sure what this has to do with an air traffic control outage either?
It does seem that a lot of infrastructure that was put in place during a golden age cannot be adequately maintained, but AFAIK this seems to be the case in USA and Europe too.
NATs has reported that the system is now back up and running. It is possible that the secondary system took over with data syncing having to take place before the switchover?
My original post was about my interest in the technical aspects of redundancy and failover mechanisms in something as mission-critical as air traffic control.
I come from a background in broadcasting, where failover is critical, and redundancy is built into any broadcast chain. e have multiple backups and jumping-off points to deal with any issues that arise. It's pretty rare we could ever go to “black”.
Whoever has the stick is responsible for the aircraft - it was a literal "unique token".
What is more impressive is how rare these events are, given the complexity of the underlying systems. Redundancy is not without its own problems (source of truth, for instance).
https://www.ft.com/content/65544730-8216-11e4-b9d0-00144feab...
And that's just the UK. Airport systems crashing is a regular occurrence. Paris crashed due to still relying on a Windows 3.1 system:
https://www.zdnet.com/article/a-23-year-old-windows-3-1-syst...
The IT is just poor. Tech firms routinely change much more massive and complex systems at a far faster pace compared to the stagnation found in airline IT, and yet total failures are not more frequent.
https://www.dw.com/en/lufthansa-system-failure-causes-massiv...
Though this is not due to outdated IT, but lack of redundancy of a critical component of the system instead.
And then after a lot of angry words and finger pointing this new failure gets added to the failure model.
My personal takeaway after chasing the long tail of automatic failover on a few projects, is quite often it is better to drop a few 9's from your service goal, decouple some of the systems, and accept that while some parts of the system may go down, it should not bring everything down with it.
https://www.wprost.pl/amp/11364934/atak-na-polska-infrastruk...
It does seem that a lot of infrastructure that was put in place during a golden age cannot be adequately maintained, but AFAIK this seems to be the case in USA and Europe too.
NATs has reported that the system is now back up and running. It is possible that the secondary system took over with data syncing having to take place before the switchover?
My original post was about my interest in the technical aspects of redundancy and failover mechanisms in something as mission-critical as air traffic control.
I come from a background in broadcasting, where failover is critical, and redundancy is built into any broadcast chain. e have multiple backups and jumping-off points to deal with any issues that arise. It's pretty rare we could ever go to “black”.