News for hackers. Stuff that matters.
Re:FAA lifted National Ground Stop at 0119Z by elicash on Friday November 13, 2026 @09:18PM (#50492971)
Here's what they said caused it in the White House briefing:

Sec Mullin: What it appears to be right now is, as Sec Duffy and FAA was going in and putting patches into the system, they were putting new equipment and new technology. And that new equipment and new technology, it was overwhelming the older system. Which is why Sec Duffy made it very clear, the patchwork doesn't work. We need one complete system.

Re:FAA lifted National Ground Stop at 0119Z by kuviaq on Friday November 13, 2026 @09:25PM (#50493026)
Re:FAA lifted National Ground Stop at 0119Z by jordanb on Friday November 13, 2026 @09:31PM (#50493092)
Seems kinda nuts to replace "obsolete" switched telephone circuit technology that's incredibly reliable and zero latency with VoIP.
Re:FAA lifted National Ground Stop at 0119Z by ajkjk on Friday November 13, 2026 @09:39PM (#50493157)
... no it doesn't?
Re:FAA lifted National Ground Stop at 0119Z by OhMeadhbh on Friday November 13, 2026 @09:40PM (#50493168)

   Sure, but the purpose of the tech policy office is to
   argue why old systems (which have fixed maintenance costs)
   should be replaced with new systems which goose the
   economy when the federal government pays 10x for what they
   should be paying.
Re:FAA lifted National Ground Stop at 0119Z by jzemeocala on Friday November 13, 2026 @09:56PM (#50493278)
reminds me of the fact that (for some reason) it costs thousands of bucks to replace or install a stop sign.....even in rural FL
Re:FAA lifted National Ground Stop at 0119Z by lelandfe on Friday November 13, 2026 @10:21PM (#50493448)
Let's say I'm a rural FL-servicing contractor the city uses. What strikes you as the right amount to pay to get a stop sign installed, cost of goods included? It's probably a two man job.
Re:FAA lifted National Ground Stop at 0119Z by mitxela on Saturday November 14, 2026 @06:42AM (#50496217)
Consider the minimum expenditure for anything is going to be a crew of at least three people for at least a full day, how would you do it for less than a thousand?
Re:FAA lifted National Ground Stop at 0119Z by xmprt on Friday November 13, 2026 @09:41PM (#50493179)
Not really. The problem with keeping old technology is that you need people who have knowledge of how to maintain the old systems. Ignoring the cynical view that many of those people were likely fired, even if they were kept on, they will eventually retire. There's no amount of training that can replace that institutional knowledge.

If and when things break you'll end up with an even bigger disaster if you don't have people prepared to maintain the old systems. A well managed decommissioning and upgrade is much more preferable.

Re:FAA lifted National Ground Stop at 0119Z by dmix on Friday November 13, 2026 @09:48PM (#50493230)
> Ignoring the cynical view that many of those people were likely fired

FAA has 45k employees. Around 2% were laid off in 2025 (around 1,600), most of which were new probationary employees or voluntary retirements. They've been hiring since then so the better question is were the subset of people who left the critical factor or some other leadership failure.

Re:FAA lifted National Ground Stop at 0119Z by sippingabonedry on Friday November 13, 2026 @10:23PM (#50493474)
> Ignoring the cynical view that many of those people were likely fired

In your mind maybe. Most are dead or retired.

New techs have been trained on splicing fiber and VoIP over the last two decades.

Nobody knows about vacuum tubes anymore either, that doesn't mean they were fired and their pensions taken away.

Re:FAA lifted National Ground Stop at 0119Z by sippingabonedry on Friday November 13, 2026 @10:12PM (#50493392)
It is obsolete.

The parts to maintain the legacy infrastructure are no longer manufactured and they are cannibalizing parts for spares.

Not to mention the expertise for these systems have either retired or in some cases passed away.

My grandparents still have a landline and the number of outages in the last 12 months has been insane. Some were in excess of a week.

Many such cases.

Re:FAA lifted National Ground Stop at 0119Z by nbaugh1 on Saturday November 14, 2026 @08:13AM (#50496768)
It’s funny, this is basically an exact analogy for the situation that my software engineering team has been in for at least 5 years now.

For various reasons the team got to a point where lots of the core services were running on totally obsolete technology. I’m talking CoffeeScript here, Python 2.7 there.

Over time, enough turnover and changes happened that they had a pile of repos where not only were the developers on the team unable/uninterested in working on the ancient tech, it was impossible to hire anyone skilled in those areas.

So we ported the features of the legacy services into a modern stack and are STILL working on getting the last of our customers migrated 2 years later. The legacy services are still up but only 2 of us really know a single thing about them, much less re-deploy them if something goes down. Documentation and run-books are great but hands on experience can be really hard to replace in the worst situations

Re:FAA lifted National Ground Stop at 0119Z by sippingabonedry on Saturday November 14, 2026 @09:18PM (#50503337)
It really isn't, and here's why.

Software environments can more easily be recreated/emulated, especially with VMs being ubiquitous.

With hardware, it's sometimes impossible to recreate. The chips are out of production. The tooling was destroyed and doesn't exist anymore. It's on a whole other level.

Re:FAA lifted National Ground Stop at 0119Z by dnqthao on Friday November 13, 2026 @09:28PM (#50493059)
I don't understand why they did not test this before deploying on production. How can we sure that when they develop a whole new system, it will work?
Re:FAA lifted National Ground Stop at 0119Z by ChrisMarshallNY on Friday November 13, 2026 @09:36PM (#50493134)
> I don't always test my code; but when I do, I do it in production.

https://static.wingify.com/gcp/uploads/abtasty/test-in-produ...

Re:FAA lifted National Ground Stop at 0119Z by OhMeadhbh on Friday November 13, 2026 @09:37PM (#50493141)

   Because if it doesn't work, they call it out for bids
   again, select their cronies and get kickbacks.  If it does
   work, then they've missed an opportunity for grift.
Re:FAA lifted National Ground Stop at 0119Z by OhMeadhbh on Friday November 13, 2026 @09:50PM (#50493242)

   Lol.  Down-voted by people who work for government
   contractors.
Re:FAA lifted National Ground Stop at 0119Z by Jtsummers on Friday November 13, 2026 @09:58PM (#50493283)
[delayed]
Re:FAA lifted National Ground Stop at 0119Z by OhMeadhbh on Saturday November 14, 2026 @12:21PM (#50498770)

   But apparently  they don't downvote people  for commenting
   on people commenting on downvoting.
Re:FAA lifted National Ground Stop at 0119Z by sippingabonedry on Friday November 13, 2026 @10:08PM (#50493365)
No you are being downvoted for abusing monospaced font for attention.
Re:FAA lifted National Ground Stop at 0119Z by dfc on Friday November 13, 2026 @10:40PM (#50493584)
They use monospace a lot. I wish there was a way to flag someone for it.
Re:FAA lifted National Ground Stop at 0119Z by mitxela on Saturday November 14, 2026 @06:41AM (#50496207)
there is, it's a button called "flag" in little gray text above the comment if you are on the comment's page.

If you meant to report the user, you have to email [email protected]. They don't bite but they don't always do the thing you wanted.

Re:FAA lifted National Ground Stop at 0119Z by OhMeadhbh on Saturday November 14, 2026 @12:20PM (#50498748)

   What about flagging people for using proportional fonts for
   attention?  Seems vaguely arbitrary.
Re:FAA lifted National Ground Stop at 0119Z by dfxm12 on Friday November 13, 2026 @09:40PM (#50493163)
This admin can't be trusted with lining a pool. Why is their incompetence still a surprise to you after all this time?
Re:FAA lifted National Ground Stop at 0119Z by dboreham on Friday November 13, 2026 @09:53PM (#50493257)
Let's set that aside and instead ask why tf they rolled out anything on a Friday? Or was it that they rolled it out on Tuesday but failed to notice it was close to overload state until it actually overloaded on Friday?
Re:FAA lifted National Ground Stop at 0119Z by kasey_junk on Friday November 13, 2026 @10:14PM (#50493403)
Or it was Friday traffic that overloaded it.
Re:FAA lifted National Ground Stop at 0119Z by golbanno on Friday November 13, 2026 @09:55PM (#50493264)
Sounds like the new system effectively performed a DoS on the old system. They probably don't have a development environment where they can hook the two systems up. Even if they do, it might actually have been an out-of-the-way component that failed first, impacting another, and so on. If they have a test environment with the two systems, a complex failure like that would unlikely to be found (and would explain why it took hours to resolve).
Re:FAA lifted National Ground Stop at 0119Z by pknomad on Friday November 13, 2026 @09:58PM (#50493292)
As someone who works under a different umbrella in the federal government, how do you propose testing something so archaic and complex without replicating an exact replica of a 40+ YO technology?
Re:FAA lifted National Ground Stop at 0119Z by AnneTrotter on Friday November 13, 2026 @10:00PM (#50493315)
I've actually worked a system where the test env was scavenged old eq set up to mirror exactly the ancient prod env. It's possible, it just takes a lot of work. Scaling it might be touchy though.
Re:FAA lifted National Ground Stop at 0119Z by pknomad on Friday November 13, 2026 @10:04PM (#50493337)
> It's possible, it just takes a lot of work

Not only it takes lot of work, but to have the replica connect to the prod in some way to mirror identical behavior will possibly mean adjusting ATO, which is not a simple process in itself.

Re:FAA lifted National Ground Stop at 0119Z by dwattttt on Friday November 13, 2026 @10:30PM (#50493528)
And it sounds like the fault conditions only occur when you do scale up.
Re:FAA lifted National Ground Stop at 0119Z by madaxe_again on Sunday November 15, 2026 @05:02AM (#50505592)
That’s obviously not true.
Re:FAA lifted National Ground Stop at 0119Z by nik282000 on Friday November 13, 2026 @10:03PM (#50493330)
Emulate, computing is cheap. In an application as important as air traffic control it seems like it wold be worth it to _always_ have a working copy of the original system specifically for testing changes.
Re:FAA lifted National Ground Stop at 0119Z by kasey_junk on Friday November 13, 2026 @10:14PM (#50493399)
You think an error that brings down _national_ networks is something you can emulate?
Re:FAA lifted National Ground Stop at 0119Z by pknomad on Friday November 13, 2026 @10:18PM (#50493430)
Thank you, you get it.
Re:FAA lifted National Ground Stop at 0119Z by cgio on Friday November 13, 2026 @10:32PM (#50493543)
Not aware of specific system, but critical infrastructure systems have a quality assurance system that replicates production albeit not perfectly. You may replicate the core system but not all its interfaces, you might replicate it but scaled down etc. There are ways for solid tests to fail. For systems with downtime, a test on production is useful iif you have solid rollback procedures. There’s lots of art in the operation of these systems.
Re:FAA lifted National Ground Stop at 0119Z by pwndByDeath on Friday November 13, 2026 @10:08PM (#50493366)
Its possible, just not probable with who you can hire for gov work.
Re:FAA lifted National Ground Stop at 0119Z by dnqthao on Friday November 13, 2026 @10:56PM (#50493724)
If you cannot replicate everything then maybe have a very conservative rollout approach so that if it fails, it only affect a small subset of the traffic.
Re:FAA lifted National Ground Stop at 0119Z by ncr100 on Saturday November 14, 2026 @07:20AM (#50496460)
That's sounding like a good idea, simulate using a RTOS.
Re:FAA lifted National Ground Stop at 0119Z by indstinctdialog on Friday November 13, 2026 @10:32PM (#50493540)
I’m sure it worked on his machine.
Re:FAA lifted National Ground Stop at 0119Z by jmyeet on Friday November 13, 2026 @11:02PM (#50493780)
The Secretary of Transportation's claim to fame is having been on MTV's the Real World and being a sycophant. Is it really that shocking? Really? We live in the dumbest timeline.

I'll be interested to see what the root cause of this all is, assuming it doesn't get buried. What I find both perplexing and predictable is how the finger has been pointed at Congress for not funding a replacement. Well, who's controlled Congress for the last 2 years? Who was president in 2017-2021? Who controlled Congress in 2017 and 2018?

If you want to dig deeper, this administration has also (unsuccessfully thus far) claimed in court that they're not bound by Congress when it comes to spending under the Impoundment Control Act.

Re:FAA lifted National Ground Stop at 0119Z by hn_throwaway_99 on Friday November 13, 2026 @09:29PM (#50493065)
I can't tell if you're being ironic or not. Sure, "one complete system", and I want a pony.

I watched Sec Duffy's presentation, and it was appalling, but par for the course with this administration: in the middle of a major nationwide outage, he tried to blame someone else.

Even if Duffy got everything he wished for, how do you think they would upgrade a system as old and complicated as ATC. They would have to do it piecemeal anyway. This was a classic upgrade that broke shit. The problem was with the new upgrade, not the old shit that worked.

Re:FAA lifted National Ground Stop at 0119Z by elicash on Friday November 13, 2026 @09:38PM (#50493151)
I was just presenting the full quote from the Secretary of Homeland Security explaining what went wrong. I wasn't commenting on it. I added quotation marks to make it more clear.

I do agree with your view.

Re:FAA lifted National Ground Stop at 0119Z by hn_throwaway_99 on Friday November 13, 2026 @09:53PM (#50493256)
Oh, my apologies, I had originally read the last 2 sentences in your voice. Thanks for the update.
Re:FAA lifted National Ground Stop at 0119Z by AndrewKemendo on Friday November 13, 2026 @09:29PM (#50493069)
This is only one of many recent “lip sores” indicating the a herpes virus that is all over US critical infrastructure.

Over a decade ago I was an intelligence officer and the alarm bells were going off and it was exceptionally clear to the National security community that US infrastructure is barely holding on.

Clapper in 2013:

https://nsarchive.gwu.edu/document/21491-document-90

Dan Coates in 2018:

https://www.euronews.com/2018/07/13/us-intel-chief-warns-of-...

Tip of the iceberg folks. Hold on to your butts…

Re:FAA lifted National Ground Stop at 0119Z by kQq9oHeAz6wLLS on Friday November 13, 2026 @10:39PM (#50493582)
But somehow it's this administration's fault.
Re:FAA lifted National Ground Stop at 0119Z by wpm on Friday November 13, 2026 @11:27PM (#50493925)
In this particular case? Yes, very much so.

Both things can be true: US infrastructure is in a dire place AND this monumental acute unforced fuck up is the fault of the unqualified, corrupt people who were put in charge of it

Re:FAA lifted National Ground Stop at 0119Z by kilroy123 on Saturday November 14, 2026 @08:15AM (#50496784)
It certainly does give strong "The Empire is Falling" vibes.
Re:FAA lifted National Ground Stop at 0119Z by drehm on Friday November 13, 2026 @09:31PM (#50493089)
Duffy also kept saying they were working as quickly as they possibly could trying to rollout the new system. Not exactly reassuring.

I think the problem with a lot of the push to "modernize" is we forget the legacy system has been real-world tested for 40 years now. It's the new stuff that needs to prove itself in terms of stability.

Re:FAA lifted National Ground Stop at 0119Z by Taek on Friday November 13, 2026 @10:14PM (#50493402)
We have a rule in my dev shop: if a piece of code (or hardware) has been working without fault in production, you do not touch it. It doesn't matter if it's old, if you don't like the tests, if it's in a programming language you despise; if there haven't been any problems in production, you don't touch it.
Re:FAA lifted National Ground Stop at 0119Z by soulofmischief on Friday November 13, 2026 @10:22PM (#50493462)
What about legitimate issues?
Re:FAA lifted National Ground Stop at 0119Z by lanyard-textile on Friday November 13, 2026 @10:26PM (#50493495)
That's the exception they noted ;)

> if a piece of code (or hardware) has been working without fault in production

Re:FAA lifted National Ground Stop at 0119Z by dwattttt on Friday November 13, 2026 @10:30PM (#50493520)
And then you have the common reality: "a piece of code has been working in production with known faults & known costs". And you have to weigh up costs and risks.
Re:FAA lifted National Ground Stop at 0119Z by Taek on Friday November 13, 2026 @11:15PM (#50493851)
Well yes of course. If there are bugs or issues in prod (performance/scale), then code can be updated or replaced.

But we have tons of legacy code that our engineers don't like, yet it runs fine and has run fine for years.

Re:FAA lifted National Ground Stop at 0119Z by Terr_ on Friday November 13, 2026 @10:32PM (#50493539)
[delayed]
Re:FAA lifted National Ground Stop at 0119Z by SoftTalker on Friday November 13, 2026 @10:45PM (#50493628)
This is exactly the way we operated Crystal Reports. "If it works, don't touch it"
Re:FAA lifted National Ground Stop at 0119Z by winrid on Sunday November 15, 2026 @08:12AM (#50506702)
Hi! I built a competitor to you for motorsports partially so I didn't have to deal with the output of Crystal Reports and have been migrating customers. Good luck tho
Re:FAA lifted National Ground Stop at 0119Z by hardaker on Friday November 13, 2026 @10:50PM (#50493674)
Especially on a Friday. The 13th doubly so.
Re:FAA lifted National Ground Stop at 0119Z by newswasboring on Friday November 13, 2026 @10:52PM (#50493688)
Sure if you don't like the tests you can write more tests, no? Tests are not production. Am I missing something?
Re:FAA lifted National Ground Stop at 0119Z by Taek on Friday November 13, 2026 @11:16PM (#50493860)
In some cases the testing frameworks cause CI headaches
Re:FAA lifted National Ground Stop at 0119Z by jiggawatts on Friday November 13, 2026 @10:54PM (#50493705)
That’s the wrong attitude.

This thinking caused an extended outage for the biggest mobile network in Australia: https://www.telstra.com.au/exchange/some-mobile-calls-and-da...

Those pair of boxes nobody was brave enough to touch cost them AUD 30 million in fines.

Re:FAA lifted National Ground Stop at 0119Z by jmyeet on Friday November 13, 2026 @11:14PM (#50493848)
I haven't read anything to suggest the root cause was similar. As far as I've seen Telstra operated in a way very much the opposite of the FAA and ATC. A reboot of a GPS module propagated the wrong time via NTP to the network. An audit said Telstra didn't treat the correct time as a critical capability (you know, with backups and such).
Re:FAA lifted National Ground Stop at 0119Z by jiggawatts on Friday November 13, 2026 @11:24PM (#50493911)
They had redundancy! Both appliances that "always worked" stopped working.

In my mind, anything that's not under continuous maintenance immediately begins to suffer from bit-rot and degrades steadily over time.

Of course, that's not a degradation of digital media, but the "dropping out" of concepts out of people's minds.

This is akin to Netflix's Chaos Monkey approach: systems under constant change and attack necessarily become robust. Systems untouched under glass like something in a museum are typically fragile and crumble if so much as breathed on.

Confidence comes from practice, not ignorance.

Re:FAA lifted National Ground Stop at 0119Z by mitxela on Saturday November 14, 2026 @06:39AM (#50496197)
This happened because the GPS module was outdated.
Re:FAA lifted National Ground Stop at 0119Z by HumblyTossed on Saturday November 14, 2026 @10:14AM (#50497676)
> That’s the wrong attitude.

Not necessarily. Debugged code is gold.

Re:FAA lifted National Ground Stop at 0119Z by treesknees on Friday November 13, 2026 @10:54PM (#50493709)
We used to do this. Now every customer and their mother is demanding upgrades because their security contractor ran a scanner that flipped out over a version number string.
Re:FAA lifted National Ground Stop at 0119Z by winrid on Sunday November 15, 2026 @08:13AM (#50506711)
The longer you don't keep things like dependencies updated, the harder it is to update the second you have a security issue etc. you can end up absolutely screwed.
Re:FAA lifted National Ground Stop at 0119Z by LastTrain on Friday November 13, 2026 @09:40PM (#50493166)
Put someone competent & accountable in charge and then let’s talk.
Re:FAA lifted National Ground Stop at 0119Z by khimaros on Friday November 13, 2026 @10:07PM (#50493350)
how could this possibly not have been SMART? were they really doing two simultaneous rollouts?
Re:FAA lifted National Ground Stop at 0119Z by bamboozled on Friday November 13, 2026 @10:21PM (#50493452)
Sounds a bit like Chernobyl. People doing stupid stuff to keep “the party” happy.
Re:FAA lifted National Ground Stop at 0119Z by mitxela on Saturday November 14, 2026 @06:47AM (#50496244)
Chernobyl was get-there-itis which affects most systems. They pushed forward when they should have rolled back.
Re:FAA lifted National Ground Stop at 0119Z by Null-Set on Friday November 13, 2026 @10:29PM (#50493513)
They also kept noting that the new IP based routers were not showing any inconsistency. To me that says that they were bridging IP traffic over the legacy systems, probably a protocol with retransmits like TCP, and a storm of retransmits was overloading the legacy systems, and older components using a less durable protocol were drowned out and dropped packets.
Re:FAA lifted National Ground Stop at 0119Z by UltraSane on Friday November 13, 2026 @10:46PM (#50493636)
Mullin doesn't actually understand anything he said.
Re:FAA lifted National Ground Stop at 0119Z by Havoc on Saturday November 14, 2026 @07:57AM (#50496668)
aka more trump crowd yolo'ing sht way out of their depth
Re:FAA lifted National Ground Stop at 0119Z by HumblyTossed on Saturday November 14, 2026 @10:12AM (#50497661)
People have been saying since the 80s that patching/piecemeal doesn't work.