Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by cratermoon on Saturday February 18, 2023 @10:59AM
(#34847841)
Concurrency is hard. Thread unsafe operations that end up getting executed in multi-threaded contexts are a huge source of heisenbugs
[ Parent
| on HN ]
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by djanogo on Saturday February 18, 2023 @11:15AM
(#34848001)
This wasn't really a concurrency problem as I understand it, the requests for calculations didn't need to share any data between them.
"the bug was missed because it only presented when many aircraft at the same time were using the system"
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by hn_throwaway_99 on Saturday February 18, 2023 @11:18AM
(#34848037)
There is certainly not enough information in that quote to say one way or another what the bug was. I've definitely seen concurrency bugs under load because data that wasn't supposed to be shared actually was, e.g. I posted this serious GitHub bug in a comment above, https://github.blog/2021-03-18-how-we-found-and-fixed-a-rare....
Obviously sessions should be independent and not sharing data, but that's why it was a bug.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by usea on Saturday February 18, 2023 @11:43AM
(#34848284)
I believe any bug that only happens under load is a concurrency bug by definition. The shared resource is the thing under load. If it weren't shared, then the load from one computation would have no effect on another.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by howinteresting on Saturday February 18, 2023 @12:08PM
(#34848560)
Should have used Rust.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by klyrs on Saturday February 18, 2023 @12:32PM
(#34848877)
A proper transactional database would suffice. You could do this safely and performantly on '00s hardware in PHP3 for crying out loud, serving thousands of planes per second.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by AdrianB1 on Saturday February 18, 2023 @04:42PM
(#34851394)
Bad coding happen even in transactional databases, see my example in the other comment.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by klyrs on Saturday February 18, 2023 @07:52PM
(#34852883)
Note that I didn't say you couldn't write bad PHP3.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by howinteresting on Saturday February 18, 2023 @10:02PM
(#34853831)
(This was a joke!)
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by speed_spread on Saturday February 18, 2023 @12:20PM
(#34848713)
I refuse to believe that summing up weights for each of 300 planes is so hard that it can't be made correct by a team of moderately competent developers operating under sane management. There might be some complexity, but it's 2023 and we have enough knowledge and tools to solve that kind of stuff routinely and reliably. Somebody fucked up big time and should be fired never to work in software again.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by lcuff on Saturday February 18, 2023 @12:45PM
(#34849035)
I agree that it's hard to see from the outside how calculating weight figures for an airplane could be software load-dependent. But this all smacks of the point that Brooks makes in 'The Mythical Man Month' all those years ago. Two guys in a garage can make a program that does 'x', but to make a programming system product that does 'x' is more than an order of magnitude more work. The complexities introduced don't relate to the 'x', but the intricacies of large systems. Deciding that 'someone has fucked up and should never work again'? Sorry, competent and conscientious people regularly make such mistakes. Which is why we need tests, software review, engineering processes et al to catch those mistakes.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by justinclift on Saturday February 18, 2023 @02:59PM
(#34850448)
Boeing itself (unsure about this company though) was widely reported to have offshored their senior engineers a few years ago. eg:
https://www.bloomberg.com/news/articles/2019-06-28/boeing-s-... (paywalled)
https://archive.is/vdg9S (paywall workaround)
Note that there were several articles about it around the time, the above one is just the first reasonable seeming one from a quick search.
Re:After Alaska Airlines planes bump runway, a scramble to ‘...
by AdrianB1 on Saturday February 18, 2023 @04:41PM
(#34851384)
I can give you a simple example on how this can happen, from a real production system in a different regulated environment: all the SQL SELECT statements are WITH (NOLOCK) on MS SQL server, resulting in non-committed (dirty) reads. Under very light load it works fine, on higher loads it reads some dirty data. If you calculate the weight of the plane by adding the weight of the checked in luggage plus passengers read from a shared database, some differences will appear and they will increase with the load as more transactions will be in flight and possibly read or ignored.