News for hackers. Stuff that matters.
Re:Deep dive into the UK air traffic control meltdown by hn_throwaway_99 on Monday September 11, 2023 @03:46PM (#37472643)
I agree with the general sentiment "if you see an unexpected error, STOP", but I don't really think that applies here.

That is, when processing a sequential queue which is what this job does, it seems to me reading the article that each job in the queue is essentially totally independent. In that case, the code most definitely should isolate "unexpected error in job" from a larger "something unknown happened processing the higher level queue".

I've actually seen this bug in different contexts before, and the lessons should always be: One bad job shouldn't crash the whole system. Error handling boundaries should be such that a bad job should be taken out of the queue and handled separately. If you don't do this (which really just entails being thoughtful when processing jobs about the types of errors that are specific to an individual job), I guarantee you'll have a bad time, just like these maintainers did.

Re:Deep dive into the UK air traffic control meltdown by Thorentis on Monday September 11, 2023 @04:09PM (#37472931)
Except that you can't be sure this bad flight plan doesn't contain information that will lead to a collision. The system needs to maintain the integrity of all plans it sees. If it can't process one, and there's the risk of a plane entering airspace with a bad flight plan, you need to stop operations.
Re:Deep dive into the UK air traffic control meltdown by lozenge on Monday September 11, 2023 @04:14PM (#37473004)
But they have 4 hours to reach out to the one plane whose flight plan didn't get processed and tell them to land somewhere else.
Re:Deep dive into the UK air traffic control meltdown by ivraatiems on Monday September 11, 2023 @04:55PM (#37473570)
Assuming they can identify that plane.

Aviation is incredibly risk-averse, which is part of why it's one of the safest modes of travel that exists. I can't imagine any aviation administration in a developed country being OK with a "yeah just keep going" approach in this situation.

Re:Deep dive into the UK air traffic control meltdown by phkahler on Monday September 11, 2023 @07:30PM (#37475078)
>> Except that you can't be sure this bad flight plan doesn't contain information that will lead to a collision.

Flight plans don't contain any information relevant for collision avoidance. They only say when and where the plane is expected to be. There is not enough specificity to ensure no collisions. Things change all the time, from late departures, to diverting around bad weather. On 9/11 they didn't have every plane in the sky file a new flight plan carefully checked against every other...

Re:Deep dive into the UK air traffic control meltdown by crabbone on Monday September 11, 2023 @05:26PM (#37473951)
> is essentially totally independent

They physically cannot be independent. The system works on an assumption that the flight was accepted and is valid, but it cannot place it. What if it accidentally schedules another flight in the same time and place?

Re:Deep dive into the UK air traffic control meltdown by krisoft on Monday September 11, 2023 @06:27PM (#37474555)
> What if it accidentally schedules another flight in the same time and place?

Flight plans are not responsible for flight separation. It is not their job and nobody uses them for that.

As a first approximation they are used so ATC doesn’t need to ask every airplane every five minute “so flight ABC123 where do you want to go today?”

I’m staring to think that there is a need for a “falsehoods programers believe about aviation” article.

Re:Deep dive into the UK air traffic control meltdown by jameshart on Monday September 11, 2023 @09:07PM (#37475777)
If the code takes a valid series of ICAO waypoints and routes, generates the corresponding ADEXP waypoint list, but then when it uses that to identify the ICAO segment that leaves UK airspace it's capable of producing a segment from before when the route enters UK airspace, then that code is wrong, and who knows what other failure modes it has?

Maybe it can also produce the wrong segment within British airspace, meaning another flight plan might be processed successfully, but with the system believing it terminates somewhere it doesn't?

Maybe it's already been processing all the preceding flight plans wrongly, and this is just the first time when this error has occurred in a way that causes the algorithm to error?

Maybe someone's introduced an error in the code or the underlying waypoint mapping database and every flight plan that is coming into the system is being misinterpreted?

Re:Deep dive into the UK air traffic control meltdown by WalterBright on Monday September 11, 2023 @09:24PM (#37475934)
An "unexpected error" is always a logic bug. The cause of the logic error is not known, because it is unexpected. Therefore, the software cannot determine if it is an isolated problem or a systemic problem. For a systemic problem, shutting down the system and engaging the backup is the correct solution.
Re:Deep dive into the UK air traffic control meltdown by iudqnolq on Tuesday September 12, 2023 @03:13AM (#37477731)
I'm pretty inexperienced, but I'm starting to learn the hard way that it takes more discipline to add more complex error recovery. (Just recently my implementation of what you're suggesting - limiting the blast radius of server side errors - meant all my tests were passing with a logged error I missed when I made a typo)

Considering their level 1 and 2 support techs couldn't access the so-called "low level" logs with the actual error message it's not clear to me they'd be able to keep up with a system with more complicated failure states. For example, they'd need to make sure that every plan rejected by the computer is routed to and handled by a human.