News for hackers. Stuff that matters.
Re:A corrupt file led to the FAA ground stoppage. It was als... by JCM9 on Thursday January 12, 2023 @08:29AM (#34353159)
The NOTAM system is something that a room full of decent engineers could easily build from scratch and make it infinitely better in a short time. It’s essentially just a database of categorized posts with some APIs for sending entries and and returning them when requested. These government IT teams spend way more than what it should cost and end up with bloated ancient tech that barely works.

That’s not speaking poor of the engineers (which in my experience can be very good) but of the management and innovation culture of these agencies, which is too often terribly broken. They would say they are “risk averse” but as yesterday highlights their poor approach to this creates a ton of risk.

Re:A corrupt file led to the FAA ground stoppage. It was als... by _fat_santa on Thursday January 12, 2023 @08:44AM (#34353368)
Well they claim to be "risk averse", the wisdom there seems to be that if you fill out a million forms that qualifies as risk averse because it shows you did due diligence. The problem is they wholesale took that process from the rest of the org and applied it to software which doesn't work.

It's like making a crap sandwich and filling out a bunch of forms proving that it's not crap sandwich, rather than just spending the time you would be filling out all those forms on i don't know...not making a crap sandwich.

Re:A corrupt file led to the FAA ground stoppage. It was als... by Someone1234 on Thursday January 12, 2023 @09:07AM (#34353642)
"I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code.

Obviously the NOTAM system isn't up to scratch, that's why we're talking about it. But "hard problems are easy" isn't constructive nor very realistic. I bet someone could put in a three-hour presentation talking about all the complexity that led us to this conclusion.

Re:A corrupt file led to the FAA ground stoppage. It was als... by JCM9 on Thursday January 12, 2023 @09:47AM (#34354218)
I hear you. But the flip side is there is often a tendency to massively over complicate things in a way that bakes in valueless complexity and bloat, which is very much the norm in large government systems. I’m practice this attitude is far more common and destructive long term.
Re:A corrupt file led to the FAA ground stoppage. It was als... by nonethewiser on Thursday January 12, 2023 @10:01AM (#34354464)
I think it's way worse than adding complexity and bloat. There are processes that specifically prevent anyone from understanding or owning the system. The complexity and bloat is a side product of the fact that the work was siloedd, contracted out, and everyone washed their hands off the result. Which also takes exponentially more time and people.

Its optimized for inefficiency.

Re:A corrupt file led to the FAA ground stoppage. It was als... by gnarbarian on Thursday January 12, 2023 @10:56AM (#34355423)
this is extremely accurate. it's like a giant rube Goldberg machine that takes both good code and garbage as input and produces a tremendous amount of garbage that nobody can understand once it comes out the other end.
Re:A corrupt file led to the FAA ground stoppage. It was als... by bink on Thursday January 12, 2023 @12:11PM (#34356926)
Do you know that each of these systems actually has a defined "owner"? It's part of the FISMA process and every so often these "owners" have to attest that certain security processes are in place and functioning and that no significant changes have been made without a full review... among many other things.

That "owner" is also a federal employee, not a contractor.

https://en.wikipedia.org/wiki/Federal_Information_Security_M...

Re:A corrupt file led to the FAA ground stoppage. It was als... by rattlesnakedave on Thursday January 12, 2023 @09:53AM (#34354312)
> This stuff is such cringe, it is the type of response you see from fresh CS students who haven't started working yet and think everything is a piece of demo work where the requirements don't matter and that everything is simple if you just start writing some code.

This stuff is such cringe, it is the type of response you see from jaded boomers more focused on box-ticking and punching out as opposed to doing anything new.

This is a situation where tossing the whole damn thing out and starting over again would be productive. The systems that lead to the creation of these half-fossilized government projects (that still don't work!) will not change and needs to be tossed.

Re:A corrupt file led to the FAA ground stoppage. It was als... by jsight on Thursday January 12, 2023 @11:07AM (#34355624)
The problem with replacing NOTAMs is that it actually is a fairly simple system that works pretty well from a technical standpoint.

The biggest issues in general relate to the data itself and the processes involved.

Re:A corrupt file led to the FAA ground stoppage. It was als... by nonethewiser on Thursday January 12, 2023 @10:00AM (#34354436)
> "I haven't seen the requirements, know little to nothing about the system, but I could knock that out in a weekend with a few Red Bulls." This stuff is such cringe

You mean the quote that you just made up?

Re:A corrupt file led to the FAA ground stoppage. It was als... by CamperBob2 on Thursday January 12, 2023 @10:03AM (#34354495)
What makes this a hard problem, exactly? It has to work no matter what, but it also doesn't have to do very much.
Re:A corrupt file led to the FAA ground stoppage. It was als... by gnarbarian on Thursday January 12, 2023 @10:59AM (#34355472)
I've worked directly with this system a number of times. it's basically a pub sub rss feed optimized for low latency and molested by bureaucrats for decades
Re:A corrupt file led to the FAA ground stoppage. It was als... by sumtechguy on Thursday January 12, 2023 @09:51AM (#34354281)
There is a whole group of people who their job depends on the system existing like it does. One dude I worked with re-did a procurement system with a few spreadsheets. He got tired of waiting for 30 people to 'do it'. As soon as his boss found it the whole project he was on was scrapped and he was demoted shortly thereafter. His 'sin'? He had accidently found a way to put 30 people out of a job. These systems exist like this because our gov wants them that way. Not because they are the best.

Yesterday should have been 'run this on these 2 old boxes and make sure they still work' regression test. Sounds like either that step is not there, totally skipped, or does not match reality. Building a real five 9s type system means taking each piece and pondering 'what are the different ways this can fail'. Then mitigating each of those. It is mind numbing tedious work than takes a long time to do. Also most of this is probably run by contract houses. Which means the people who use it do not really 'own it'. Which is by design, for CYA. Which costs more and takes more time because it is all paperwork.

Re:A corrupt file led to the FAA ground stoppage. It was als... by nonethewiser on Thursday January 12, 2023 @09:57AM (#34354392)
And the kicker is that you're paying for it.
Re:A corrupt file led to the FAA ground stoppage. It was als... by phaedrus on Thursday January 12, 2023 @11:20AM (#34355863)
One common problem is building new things with tech that is already obsolete. Given the choice between new and old, the old is perceived as more tried and tested. Granted it's sometimes difficult to distinguish "new and going to last" from "new and shiny", but when it comes to software choosing to go with "old and tested" can be counter to cyber security since the old software is no longer updated.

Of course you also have "old but still developed, and likely to be supported in near-perpetuity". Things like SQLite fall in this category. Unfortunately there's this other problem of management being inexplicably down on anything open-source. I don't know whether they've been exposed to too much FUD from vendors selling proprietary solutions, or just that the idea that security through obscurity is no security at all has failed to reach them.

Re:A corrupt file led to the FAA ground stoppage. It was als... by killerstorm on Friday January 13, 2023 @01:22PM (#34371749)
Engineers following current best practices? They'd run a Kubernetes cluster with Kafka. Because those are the best CotS tool for reliability. Battle-tested. The system will be down every week because Kubernetes need patching.