News for hackers. Stuff that matters.
Re:Deep dive into the UK air traffic control meltdown by cratermoon on Sunday September 10, 2023 @10:53PM (#37462318)
> why could the system not put the failed flight plan in a queue

Because it doesn't look at the data as a "flight plan" consisting of "way points" with "segments" along a "route" that has any internal self-consistency. It's a bag of strings and numbers that's parsed and the result passed along, if parsing is successful. If not, give up. In this case fail the entire systemand take it out of production.

Airline industry code is a pile of badly-written legacy wrappers on top of legacy wrappers. (Mostly not including actual flight software on the aircraft. Mostly). The FPRSA-R system mentioned here is not a flight plan system, it's an ETL system. It's not coded to model or work with flight plans, it's just parsing data from system A, re-encoding it for system B, and failing hard if it it can't.

Re:Deep dive into the UK air traffic control meltdown by slt2021 on Monday September 11, 2023 @12:43PM (#37469901)
good ETLs are usually designed to separate good records from bad records, so even if one or two rows in the stream do not conform to schema - you can put them aside and process the rest.

seems like poor engineering

Re:Deep dive into the UK air traffic control meltdown by jandrese on Monday September 11, 2023 @01:43PM (#37470786)
The problem is that it means you have a plane entering the airspace at some point in the near future and the system doesn't know it is going to be there. The whole point of this is to make sure no two planes are attempting to occupy the same space at the same time. If you don't know where one of the planes will be you can't plan all of the rest to avoid it.

The thing that blows my mind is that this was apparently the first time this situation had happened after 15 million records processed. I would have expected it to trigger much more often. It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day.

Re:Deep dive into the UK air traffic control meltdown by d1sxeyes on Monday September 11, 2023 @03:00PM (#37472047)
> It makes me wonder if there wasn't someone who was fixing these as they came up in the 4 hour window, and he just happened to be off that day.

This is very possible. I know of a guy who does (or at least a few years ago did) 24x7 365 on-call for a piece of mission (although not safety) critical aviation software.

Most of his calls were fixing AWBs quickly because otherwise planes would need to take off empty or lose their take-off slot.

Although there had been some “bus factor” planning and mitigation around this guy’s role, it involved engaging vendors etc. and would have likely resulted in a lot of disruption in the short term.

Re:Deep dive into the UK air traffic control meltdown by fbdab103 on Monday September 11, 2023 @04:34PM (#37473300)
Please tell me this guy is now wealthy beyond imagination and living a life of leisure?
Re:Deep dive into the UK air traffic control meltdown by d1sxeyes on Tuesday September 12, 2023 @03:30AM (#37477853)
I would love to. But it wouldn’t be true.
Re:Deep dive into the UK air traffic control meltdown by zaphar on Monday September 11, 2023 @03:12PM (#37472239)
Bad records aren't supposed to be ignored. They are supposed to be looked at by a human who can determine what to do.

Failing the way NATS did means that all future flight plan data including for planes already in the sky are not longer being processed. The safer failure mode was definitely to flag this plan and surface to a human while continuing to process other plans.

Re:Deep dive into the UK air traffic control meltdown by epolanski on Monday September 11, 2023 @07:46PM (#37475202)
One in a 15M chance with 7000 daily flies over the UK handled by nats meant it had a probability to happen at least once in 69 months, it took few months less.
Re:Deep dive into the UK air traffic control meltdown by cratermoon on Monday September 11, 2023 @05:25PM (#37473927)
I never said it was a good ETL system. Heck, I don't even know if the specs for it even specifies what to do with a bad record - there are at least 300 pages detailing the system. Looking around at other stories, I see repeated mentions of how the circumstances leading to this failure are supposedly extremely rare, "one in 15 million" according to one official[1]. But at 100,000 flights/day (estimated), this kind situation would occur, statistically, twice a year.

1 https://news.sky.com/story/major-flights-disruption-caused-b...

Re:Deep dive into the UK air traffic control meltdown by Scarblac on Monday September 11, 2023 @05:52PM (#37474214)
This flight plan was correct though, if there was some validation like that then it should have passed.

The code that crashed had a bug, it couldn't deal with all valid data.