News for hackers. Stuff that matters.
Re:FAA Issues National Ground Stop by sweswas2 on Friday November 13, 2026 @07:00PM (#50491941)
Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday? This would be extremely unusual in any software job I have ever had. For commercial aviation specifically which does have pretty well defined peaks and valleys in terms of traffic over the course of a day this seems like an odd time to choose
Re:FAA Issues National Ground Stop by mainmailman on Friday November 13, 2026 @07:05PM (#50491978)
Maybe patching a critical vulnerability? I’m also at a loss for why they chose Friday afternoon.
Re:FAA Issues National Ground Stop by dietr1ch on Friday November 13, 2026 @07:08PM (#50492001)
I'd wager management wanted this by the EoW
Re:FAA Issues National Ground Stop by a012 on Friday November 13, 2026 @07:37PM (#50492214)
RCA could be in just 2 words: incompetent people
Re:FAA Issues National Ground Stop by nik282000 on Friday November 13, 2026 @07:37PM (#50492219)
Weird how all epic stupid decisions come from the people who are supposedly the trained 'decisions makers.'

I spend at least 1/3 of my fridays answering all requests with "Read Only Friday."

Re:FAA Issues National Ground Stop by EvanAnderson on Friday November 13, 2026 @07:52PM (#50492312)
Arbitrary date-based deadlines piss me off so much.

I worked for a person who had that mentality-- the "first of the month", "the end of the week", "the start of the quarter", etc. It would be one thing if these dates were tied to legal compliance (tax filings, etc), but that usually wasn't the case.

I'd be sure to time meetings to maximize time for work-- meet late Wednesday because an early Tuesday meeting might get me an "end of the week" deadline, but a late Wednesday meeting might get be "the end of next week". You always had to be aware of the numeric calendar day, too. Gunning for an "end of next week" deadline slot when next Wednesday was the end of the month (or worse, end of the quarter) almost guaranteed you an earlier deadline than you wanted.

The amount of time the work was going to take had no bearing on it.

It was also immensely fun, when setting a deadline in this person's absence, to get a group to agree on a mid-week deadline. You could see it make this person's skin crawl hearing that the work would be done on Tuesday or a Thursday.

Re:FAA Issues National Ground Stop by tantalor on Friday November 13, 2026 @11:36PM (#50493987)
"The project will land on the first full moon after the vernal equinox"
Re:FAA Issues National Ground Stop by gregw2 on Saturday November 14, 2026 @07:59AM (#50496686)
"More testing? We need to get this out! We need a win!"
Re:FAA Issues National Ground Stop by cjbprime on Friday November 13, 2026 @08:03PM (#50492396)
That could be their least busy day, for all we know. Weekends are busier than weekdays for airline travel.
Re:FAA Issues National Ground Stop by alistairSH on Friday November 13, 2026 @08:54PM (#50492782)
Friday and Sunday are the busiest days. Business and pleasure overlap. Sat is mostly pleasure. Mon-Thu are mostly business.
Re:FAA Issues National Ground Stop by ethbr1 on Saturday November 14, 2026 @05:50AM (#50495943)
Tuesday and Wednesday are apparently the least busy days, at average -10% traffic each.

So the deployment target seems like it should have been Monday @ 10pm. (Or whenever the bulk of flights have landed, but there's still enough for a validation test)

Re:FAA Issues National Ground Stop by mrandish on Friday November 13, 2026 @09:20PM (#50492987)
But 3am is when a big chunk of planes are parked at gates awaiting the start of the flight day (yes, there are red-eye flights and international but they are a minority.)

Overnight periods are when the air travel system largely resets itself from each day's cascading delays, mechanical failures and bad weather.

Re:FAA Issues National Ground Stop by adgjlsfhk1 on Saturday November 14, 2026 @12:02AM (#50494133)
9pm is likely better. if you patch at 3am there's a decent chance no one notices until 6 and then you ruin the next day.
Re:FAA Issues National Ground Stop by Melatonic on Saturday November 14, 2026 @02:48AM (#50494958)
It's also very possible they patched last night since they were possible also updating systems for the big AI rollout and then the issues didn't cascade or cause issues until this afternoon
Re:FAA Issues National Ground Stop by salt-thrower on Friday November 13, 2026 @07:07PM (#50491998)
I did web dev work for an airline and we couldn't even release new updates to the loyalty benefits website during a weekday (even though there was no connection to operations or flights). The fact that this company would reboot a critical comms server at 3 pm on a Friday is indeed bizarre.
Re:FAA Issues National Ground Stop by NooneAtAll5 on Friday November 13, 2026 @07:24PM (#50492126)
> we couldn't even release new updates during a weekday

so you were forced to release updates on weekends? isn't that worse?

Re:FAA Issues National Ground Stop by margalabargala on Friday November 13, 2026 @07:31PM (#50492157)
Worse for the employees. Better for the clients.
Re:FAA Issues National Ground Stop by gdavisson on Friday November 13, 2026 @07:35PM (#50492189)
It depends on the schedules of the people who'll have to fix it if something goes wrong. I don't mind working weekends, so when I was in the sysadmin game I always tried to make significant changes late friday/early saturday, so if something went wrong I had the weekend to fix it & clean up. But if the people needed to fix a problem take weekends off, pretty much the opposite schedule applies.
Re:FAA Issues National Ground Stop by 80fj0348i on Friday November 13, 2026 @07:46PM (#50492278)
Flights generally peak Mon-Thu due to business flights, weekends are usually lighter unless it's a long holiday weekend.
Re:FAA Issues National Ground Stop by luxuryballs on Friday November 13, 2026 @07:49PM (#50492293)
the update caused the reboot but could that mean the reboot happened due to a prior update? or literally the reboot was FOR the update hmm
Re:FAA Issues National Ground Stop by fy20 on Friday November 13, 2026 @11:58PM (#50494100)
Windows Update decided to reboot automatically because somebody didn't press the skip button enough time?
Re:FAA Issues National Ground Stop by ransom1538 on Friday November 13, 2026 @08:00PM (#50492379)
"3 pm on a Friday is indeed bizarre"

Most likely, they HAD to do it.

Re:FAA Issues National Ground Stop by wombatpm on Friday November 13, 2026 @08:05PM (#50492413)
Probably was scheduled for Thursday evening but was delayed due to the Outlook issues.
Re:FAA Issues National Ground Stop by ocdtrekkie on Friday November 13, 2026 @08:41PM (#50492689)
IT in corporate is actually more panicked than IT in life safety, because life safety invests more redundant design.
Re:FAA Issues National Ground Stop by Rapzid on Friday November 13, 2026 @08:47PM (#50492718)
I believe not deploying on Friday is an anti-pattern for most software teams.
Re:FAA Issues National Ground Stop by luipugs on Friday November 13, 2026 @09:06PM (#50492886)
What industries have you seen teams deploy on a Friday afternoon?
Re:FAA Issues National Ground Stop by mlhpdx on Friday November 13, 2026 @09:12PM (#50492922)
It happens. My company has done a couple dozen production rollouts today, including minutes ago.
Re:FAA Issues National Ground Stop by saturn8601 on Friday November 13, 2026 @09:38PM (#50493154)
Uh I'd imagine any team that mainly has their user base using the product on weekdays would deploy Friday evening so if something goes wrong they have the weekend to fix it.
Re:FAA Issues National Ground Stop by paulhebert on Friday November 13, 2026 @09:55PM (#50493268)
Only if the team wants to work weekends?

If something goes down on Saturday there’s a good chance I’m out of cell service

Re:FAA Issues National Ground Stop by shadowgovt on Friday November 13, 2026 @11:50PM (#50494055)
[delayed]
Re:FAA Issues National Ground Stop by saturn8601 on Saturday November 14, 2026 @12:48AM (#50494383)
I mean, you make it sound like releases fail often. For our team I think we have only seen one failure in the last two years needing a fix after hours (going into the weekend)?

The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.

Re:FAA Issues National Ground Stop by cowlevel on Saturday November 14, 2026 @01:26AM (#50494562)
Did you know that when you use your 2 days off to go shopping, someone has to work at the shops?
Re:FAA Issues National Ground Stop by shadowgovt on Saturday November 14, 2026 @10:24AM (#50497759)
[delayed]
Re:FAA Issues National Ground Stop by cowlevel on Saturday November 14, 2026 @11:18AM (#50498212)
And part of that is software engineers doing deployments after-hours for their customers. If your company or country is any good, they'll give you double time off during normal hours.
Re:FAA Issues National Ground Stop by hdgvhicv on Saturday November 14, 2026 @04:41AM (#50495553)
Many people work shifts, it’s quite normal.

What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.

Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!

Re:FAA Issues National Ground Stop by saturn8601 on Saturday November 14, 2026 @12:52AM (#50494404)
Such a smug privileged answer given in a world where the industry is shrinking its workforce and pretty much all other industries have flatlined for years(meaning we are still better off despite this realities).
Re:FAA Issues National Ground Stop by paulhebert on Saturday November 14, 2026 @11:09AM (#50498137)
Wanting to have a weekend is smug and privileged now?

People fought hard for the 40 hour work week and 2 day weekend.

You think that should be given up?

Re:FAA Issues National Ground Stop by saturn8601 on Saturday November 14, 2026 @06:43PM (#50502280)
Oh, I know all about that history. Having been a volunteer for Bernie Sanders back in 2015, I admired him as being one of the remaining few people who actually remembered this history due to him being part of the Silent Generation and continue to fight for it decades later.

>You think that should be given up?

Not really what I said. My point was that in an era of transition where we started with a 40 hour workweek and a two day weekend, the positive outcome could be a 30 hour workweek and a three day weekend or more. That transition period from one to the other is extremely precarious for everybody and is what we are in currently. It isn't going to transition until a great deal of people make it so, and they only do that when the pain reaches home for them. In the interim, your advice is counterproductive. My take away from this past election: the overwhelming majority are not going to change until it gets real for them. Your advice is to light yourself on fire in the meantime.

Re:FAA Issues National Ground Stop by paulhebert on Saturday November 14, 2026 @07:25PM (#50502593)
I really don’t understand how you got that from my post.

I mentioned the fact that I’m often out of cell service on weekends (I like to hike and camp) and so it’s a bad time for an outage.

Are you saying labor conditions need to get worse for everyone in order to get better?

If so I disagree and think we should be advocating for better labor conditions for everyone while protecting things like weekends

Re:FAA Issues National Ground Stop by Geezus_42 on Friday November 13, 2026 @09:57PM (#50493279)
We do for clients who have 24/7 users during the week.
Re:FAA Issues National Ground Stop by selcuka on Friday November 13, 2026 @10:07PM (#50493359)
I don't think the hesitation to push to prod Friday afternoons has anything to do with the working schedule of your customers. It's more about whether you'll be able to roll back on a weekend.
Re:FAA Issues National Ground Stop by Geezus_42 on Friday November 13, 2026 @10:19PM (#50493437)
Why wouldn't you be able to rollback on a weekend like any other day?
Re:FAA Issues National Ground Stop by schiffern on Friday November 13, 2026 @11:11PM (#50493825)
Not a family man, I see?
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:28AM (#50496497)
That has nothing to do with it. I don't like working weekends anymore that anyone else. However, every now and then it is necessary for certain systems. Also, if your rollback procedure is go, it isn't issue.
Re:FAA Issues National Ground Stop by schiffern on Saturday November 14, 2026 @09:39AM (#50497372)

  >I don't like working weekend anymore [sic] that [sic] anyone else.
So then why the feigned confusion? It sounds like you already knew the answer to your 'question.'
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @05:19PM (#50501559)
My confusion is people acting like it is blasphemous to deploy on a Friday when I have laid at multiple reasons why it may be necessary.
Re:FAA Issues National Ground Stop by latentsea on Friday November 13, 2026 @11:20PM (#50493887)
You can, but it also means you might have to.
Re:FAA Issues National Ground Stop by ezfe on Friday November 13, 2026 @11:20PM (#50493889)
cause I don't want to work on the weekend. Why would I introduce that risk?
Re:FAA Issues National Ground Stop by kevin_nisbet on Saturday November 14, 2026 @02:13AM (#50494778)
Or if you need more resources, can be a lot harder on a friday evening or weekend to get ahold of someone else to help.
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:21PM (#50498766)
Assuming they don’t have an actual life and aren’t somewhere touching grass.

I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.

Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.

Re:FAA Issues National Ground Stop by dzhiurgis on Saturday November 14, 2026 @02:56AM (#50495002)
It’s ok, claude can do it for you.
Re:FAA Issues National Ground Stop by Rapzid on Saturday November 14, 2026 @05:09AM (#50495692)
Thursday is pretty close to the weekend. Too risky on Thursday?
Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Saturday November 14, 2026 @06:22AM (#50496117)
I'm old, I do prod changes Monday to Wednesday for that reason. I'm not paid to work on weekends. Sure, you can argue that people should be competent enough to never release software with a bug, but I sometimes fuck up.
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:30AM (#50496509)
Or just have a good rollback plan like a snapshot.
Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Saturday November 14, 2026 @08:57AM (#50497065)
If the system I work on needs to be restored from a snapshot then I'll definitely be spending the weekend working. There are systems you can't just rollback and restore a snapshot and all is fine.
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @09:30AM (#50497291)
So when you roll a patch out on a Tuesday morning and it breaks, how much revenue do your client's lose? I'm all for telling a client to eat it because I'm not working on the weekend for some BS, but there are critical systems that CAN'T be offline during the week.
Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Sunday November 15, 2026 @06:26AM (#50506095)
Me and my colleagues don't want to work weekends, so that goes into the planning of deployments. There is always a possibility of failure when releasing software. I don't see the "how much revenue" relevant to what I wrote - if we deploy Friday and it breaks, we need to work weekend.
Re:FAA Issues National Ground Stop by Geezus_42 on Sunday November 15, 2026 @08:02AM (#50506641)
We don't WANT to work on weekends either. That's why it is extraordinarily rare that we do. Plus the fact that most clients have a Dev environment that we test the changes in first.

Clients losing revenue because you broke their application mid-week is important because contributes to your attrition rate. Which, if it's bad enough, will end your company.

Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Sunday November 15, 2026 @08:56AM (#50506973)
I don't know, it feels like you're arguing based on your knowledge about the system you work on, without taking into account that other systems does not work the same. It's just nonsense. Good luck.
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:33PM (#50498919)
And then there’s the Tower of Hanoi of breaking major changes into tiny pieces that can increment without making rollbacks impossible. Always two more steps than you would think you need and requires patience that management often doesn’t possess.

Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.

If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.

Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.

Re:FAA Issues National Ground Stop by ndsipa_pomu on Saturday November 14, 2026 @10:20AM (#50497726)
Thursday is a good choice, especially if Fridays are less busy than other days as that gives the team a day to work on the issue if necessary and minimises the business impact if it goes bang.
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:26PM (#50498832)
Scrum originally recommended people run sprints from Wednesday to Wednesday so you were doing deployments in the middle of the week. I’ve never worked anywhere that did a Scrum right, and I don’t mean that in the “well then you’re doing it wrong” sense. I used Kanban before Scrum and man is that a downgrade.
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:29AM (#50496504)
Because you are patching a critical system that can't be down during the week.
Re:FAA Issues National Ground Stop by ndsipa_pomu on Saturday November 14, 2026 @10:18AM (#50497713)
In which case, why wouldn't you deploy it during the weekend? That way, you're already prepared to be working several hours on a Saturday (presumably earning overtime rates) and so it's no major issue to rollback if it goes wrong.

Friday deployments make no sense if you have teams that work Monday-Friday.

Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:24PM (#50498805)
[delayed]
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @05:22PM (#50501591)
Why do you only have a single co-worker that knows that error? Why hasn't anyone written shared documentation?
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @06:19PM (#50502108)
[delayed]
Re:FAA Issues National Ground Stop by Geezus_42 on Sunday November 15, 2026 @07:59AM (#50506621)
We host a private cloud, so the only code we write are occasional shell scripts based on repeatable steps that have already been done manually hundreds or even thousands of times. And yeah, everyone on the team has a pretty thorough knowledge of the applications we host. Documentation is usually enough to fill the blanks.

I'm not saying surprises don't happen, just that they should be extraordinarily rare.

Re:FAA Issues National Ground Stop by hinkley on Sunday November 15, 2026 @05:30PM (#50511556)
I think when we are discussing a very public outage though, we can take it as read that there are any number of problems with the company and this is just the one they got caught out on. And similarly that a lot of replies are from people who have experienced the same with other businesses and/or employers.

As long as you guys don't get underbid by a competitor with a severe case of Dunning Kruger, which I've seen happen too often, then you're good. Usually you have to sell your competence as part of your story. Because you'll never match idiots on price.

Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @05:21PM (#50501573)
Friday afternoon is not unusual if you are facing a weekend deployment. Might as well try to get it done Friday afternoon an not have to worry about it the rest of the weekend.
Re:FAA Issues National Ground Stop by DANmode on Friday November 13, 2026 @10:40PM (#50493589)
or want to have to.
Re:FAA Issues National Ground Stop by james_marks on Saturday November 14, 2026 @10:04AM (#50497591)
We avoided Friday deploys because we didn’t have enough usage over the weekend; traffic was oriented on M-F working hours.

It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.

Much better to get fast feedback while the deploy could be monitored against prod traffic.

Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:18PM (#50498733)
We did Monday morning IST, on a build we declared valid on Friday afternoon - no pressure to declare premature victory there. The cherry on top was sprint demos first thing in the morning PST so when I first joined the project the demos got interrupted on a regular basis.

I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.

Re:FAA Issues National Ground Stop by pferde on Saturday November 14, 2026 @04:37AM (#50495533)
So... it's 24/5, then.
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:30AM (#50496510)
Yes pendant
Re:FAA Issues National Ground Stop by hdgvhicv on Saturday November 14, 2026 @04:37AM (#50495535)
We have systems who’s main use in at the weekend. Other systems are only used overnight. Some the quietest part is 0930-1230 on weekdays.

The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.

Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:33AM (#50496534)
There is someone on call at all times at every level in my company and we are not very big. It's rare I see anything I can't fix, but if I do, it's easy to click that button that makes the alarms go off and they'll be on the call in minutes.
Re:FAA Issues National Ground Stop by fn-mote on Saturday November 14, 2026 @12:21PM (#50498761)
Is the guy on call your best troubleshooter? No. That guy doesn’t work weird hours because you’d have to be paying him $50k more to put up with that BS.

It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.

Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @05:30PM (#50501663)
Actually, we ALL work on call. No one gets out of it just because they have seniority. Plus, we have an excellent team of smart people who are able to figure most things out on their own through use of their own knowledge and troubleshooting skills, aided by both internal and external documentation. It is rare that escalation is required.

I never said things don't go to hell sometimes. That's why you build a good team and train your people.

Re:FAA Issues National Ground Stop by asveikau on Saturday November 14, 2026 @04:50PM (#50501289)
I think it's not just inconvenience for users but also availability of engineers to look at failures.

Many organizations do have round the clock staff but maybe they have the most staff during 9-5 on a particular time zone. So end of day Friday is guaranteeing you'll get a different shift of worker diagnosing failures, i.e. not the author of the deployed change, and possibly fewer available people.

Re:FAA Issues National Ground Stop by ElProlactin on Friday November 13, 2026 @10:34PM (#50493552)
Team America: World Police
Re:FAA Issues National Ground Stop by camcil on Friday November 13, 2026 @11:35PM (#50493983)
In public utility we only did deployments on Fridays after 5PM
Re:FAA Issues National Ground Stop by shadowgovt on Friday November 13, 2026 @11:49PM (#50494046)
[delayed]
Re:FAA Issues National Ground Stop by hunter2_ on Saturday November 14, 2026 @12:30AM (#50494296)
When the update requires downtime (or has such a high likelihood of downtime) that you'd be knowingly (or almost knowingly) inconveniencing users, do it off-hours (e.g. nights and weekends) and gather availability commitments from all the people you anticipate might be useful. If something goes so terribly sideways that it takes a long while to stabilize given the skeleton crew, at least you have plenty of time before users need it again. Friday evening maximizes that window.

Else, do it during regular business hours and very far from time off (whether that's CoB, weekend, PTO, etc. -- "not on Friday"). If something goes so terribly sideways that users are impacted, at least you have nearly everyone available so hopefully it won't take long to stabilize.

Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:00PM (#50498565)
> Friday evening maximizes that window.

For the FAA it sure as fuck doesn’t. You a bot?

Re:FAA Issues National Ground Stop by sgarland on Saturday November 14, 2026 @07:27AM (#50496496)
Unless you have god-tier CI/CD, which most places do not, I do not like Friday changes. I’ve worked at precisely one place where it didn’t bother me, and one place where they didn’t have CD, but shared my belief and didn’t deploy on Friday.
Re:FAA Issues National Ground Stop by thephyber on Friday November 13, 2026 @07:24PM (#50492118)
Perhaps.

But the entire first months of the Iran War was started on Friday after the markets closed. It took a few weeks for most Americans to catch on.

My money is on some mundane ops issue like "it takes lots of time to get sign off for the upgrade from lots of different stakeholders and the necessary upgrade was falling days or weeks behind, leading to a key IT person to decide to YOLO the upgrade/deployment".

I've heard that Sec Transport Duffy has been pushing FAA upgrades faster than is safe, so maybe the employee rushed it so this upgrade wouldn't hold back some other larger timeline. There have been several articles recently about how the FAA is full of people who are "too risk averse", suggesting that there might be an effort to push riskier projects faster. Having a nationwide outage is one of the likely symptoms of raising risk tolerance.

Re:FAA Issues National Ground Stop by panarky on Friday November 13, 2026 @07:34PM (#50492185)
My money is on DNS, but if it is enemy action, will the FAA even know that's what it is?
Re:FAA Issues National Ground Stop by teachrdan on Friday November 13, 2026 @07:50PM (#50492301)
They will when the enemy -- presumably the IRGC -- takes credit for it.
Re:FAA Issues National Ground Stop by hydrox24 on Friday November 13, 2026 @07:54PM (#50492333)
> It took a few weeks for most Americans to catch on

In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_.

Re:FAA Issues National Ground Stop by asdfman123 on Friday November 13, 2026 @07:27PM (#50492136)
> Anybody familiar with these systems know why they would do an update like this at 3pm on a Friday

As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.

Yes, this comment is tongue in cheek, but not everyone follows the "don't deploy on Friday" rule, even people in mission critical fields.

Re:FAA Issues National Ground Stop by takeda on Friday November 13, 2026 @07:31PM (#50492166)
> I would surmise the technical reason for it is that they didn't want to work over the weekend.

Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend.

Re:FAA Issues National Ground Stop by asdfman123 on Friday November 13, 2026 @07:43PM (#50492257)
That's why you don't deploy on a Friday. :)
Re:FAA Issues National Ground Stop by ExoticPearTree on Friday November 13, 2026 @09:59PM (#50493304)
There is also a real possibility they did this many times before and it worked. I don’t really get it: if the thing would have broken on a Monday night and it would take a day to fix would that have been better?
Re:FAA Issues National Ground Stop by lazide on Friday November 13, 2026 @11:47PM (#50494037)
The odds of someone screwing something up just before the weekend are way higher. Both because it’s obviously a rush job (it can’t wait until Monday), and because it’s the end of the week and folks are tired.

It’s just dumb.

Re:FAA Issues National Ground Stop by throw0101d on Saturday November 14, 2026 @07:32AM (#50496527)
> As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.

Are they unionized? Maybe management didn't want to pay time-and-a-half(+) for weekend work?

Re:FAA Issues National Ground Stop by _heimdall on Friday November 13, 2026 @07:41PM (#50492239)
Nothing like a good Friday afternoon did lot to ruin a weekend.
Re:FAA Issues National Ground Stop by vlovich123 on Friday November 13, 2026 @08:18PM (#50492506)
A security patch would be a reason not to wait.
Re:FAA Issues National Ground Stop by Rapzid on Friday November 13, 2026 @08:49PM (#50492738)
And because a security patch can't wait no deployment should wait because it's Friday. That's how you stay ready; you're always ready.
Re:FAA Issues National Ground Stop by maerF0x0 on Friday November 13, 2026 @08:24PM (#50492550)
It should be fine to update at any moment. Why? Because systems like this ought to be hot spare or blue/green models that have a full working copy at any given time, and a failover/failback.
Re:FAA Issues National Ground Stop by Waterluvian on Friday November 13, 2026 @08:34PM (#50492628)
A reasonable deployment plan considers the reality that deployment could fail. One mitigation is to pick the ideal time and place in case it fails.
Re:FAA Issues National Ground Stop by bombcar on Friday November 13, 2026 @08:36PM (#50492642)
You can also have reasonable updates done at reasonable times that don’t show issues until unreasonable hours - or something else fails.

But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.

Re:FAA Issues National Ground Stop by ExoticPearTree on Friday November 13, 2026 @09:46PM (#50493218)
As opposed to what alternative? Being run by the FAA itself or with contractors? Being run by the DOT?

Well, it worked so far, I think it is great. 100% availability is a dream.

That’s why for example, you have ATC Zero protocols when communications go out all of a sudden at airports. And I imagine there is something similar for airborne planes that are not near any of their destinations.

Re:FAA Issues National Ground Stop by vikramkr on Friday November 13, 2026 @10:28PM (#50493509)
Yeah either the FAA or the DOT sounds good. Not contractors tho.
Re:FAA Issues National Ground Stop by SubmarineClub on Friday November 13, 2026 @11:21PM (#50493891)
you seriously think the government would do a better job?
Re:FAA Issues National Ground Stop by esseph on Friday November 13, 2026 @11:53PM (#50494075)
[delayed]
Re:FAA Issues National Ground Stop by uoaei on Saturday November 14, 2026 @01:35AM (#50494595)
This is a narrow and shortsighted take. Government would absolutely do a better job if only for the SOPs around redundancy that business is famously flexible on (read: happy to ignore).
Re:FAA Issues National Ground Stop by Vvector on Saturday November 14, 2026 @08:26AM (#50496837)
the government would do a better job than a private company with incentives to maximize profits.
Re:FAA Issues National Ground Stop by vikramkr on Saturday November 14, 2026 @02:13PM (#50499861)
No. I think the fact that the government wouldn't do a better job is a skill issue and the government has to get good. But also this government where it does farm out to contractors and monopolizte for every little thing could never build the software itself. Even saying "the government makes it itself" is saying we fundamentally have a government tha is different than right now.

It's been pretty cool to see NASA lock in under the new administrator, starr making things in house and not wait on contractors etc.

Re:FAA Issues National Ground Stop by snypher on Friday November 13, 2026 @11:00PM (#50493752)
I mean they also make the CPU for the F-35, alongside kinetic strike and electronic warfare missiles, so they could cause plenty of damage if they wanted to. This isn't some mom-and-pop radio company.
Re:FAA Issues National Ground Stop by heartbreak on Saturday November 14, 2026 @04:44AM (#50495572)
Based on what little I know of L3Harris, they are a pretty important part of a lot more than ATC.
Re:FAA Issues National Ground Stop by RetroTechie on Saturday November 14, 2026 @06:46AM (#50496243)
> But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.

Foreign cyber-adversaries (the usual suspects ;) are taking notes.

Re:FAA Issues National Ground Stop by sweswas2 on Friday November 13, 2026 @08:36PM (#50492648)
I mean sure and we have that in finance but there’s no amount of time saved that could make me deploy something during market hours
Re:FAA Issues National Ground Stop by logifail on Friday November 13, 2026 @08:46PM (#50492715)
Airspace certainly has busy and quiet(er) periods, but the skies are open all day every day.
Re:FAA Issues National Ground Stop by uoaei on Saturday November 14, 2026 @01:36AM (#50494604)
There are obviously crests and troughs in flight volume (as measured in passenger-hours), 3pm on a Friday seems pretty cresty to me.
Re:FAA Issues National Ground Stop by logifail on Saturday November 14, 2026 @01:47PM (#50499626)
> 3pm on a Friday seems pretty cresty to me

It wouldn't be my pick, that's for sure.

Re:FAA Issues National Ground Stop by skort on Friday November 13, 2026 @08:40PM (#50492680)
Well clearly they don't have a hot spare or blue/green models, or working failover.
Re:FAA Issues National Ground Stop by monkpit on Friday November 13, 2026 @08:56PM (#50492804)
Which is why deploying at 3pm on a Friday is even more bizarre.
Re:FAA Issues National Ground Stop by ajsnigrutin on Friday November 13, 2026 @10:30PM (#50493527)
Sure, but if it fails and a fallback takes over, will you be going home and having a nice weekend? Or will you be stuck at work over the weekend to fix it?
Re:FAA Issues National Ground Stop by tantalor on Friday November 13, 2026 @11:34PM (#50493973)
I'd much rather the teams' weekend be impacted than millions of travelers.
Re:FAA Issues National Ground Stop by lazide on Friday November 13, 2026 @11:45PM (#50494026)
What if you could avoid the team impact and not screw millions of travelers.
Re:FAA Issues National Ground Stop by ajsnigrutin on Saturday November 14, 2026 @12:00AM (#50494116)
Why not do a deploy on a monday morning, and impact the teams workday instead of their weekend?
Re:FAA Issues National Ground Stop by stridTastic on Friday November 13, 2026 @08:38PM (#50492664)
Rolling out updates on a Friday afternoon is still practiced in various industries.
Re:FAA Issues National Ground Stop by ExoticPearTree on Friday November 13, 2026 @09:57PM (#50493281)
> Rolling out updates on a Friday afternoon is still practiced in various industries.

By people with unreliable systems and a lot of myhts behind.

In a well engineered organization, you can deploy whenever you want. Sure, it will not work 100% of the time, but you figure out why, fix the problem and you will be one step closer to the mythical 100%. And so on. Deploying less frequently is what creates more problems.

Re:FAA Issues National Ground Stop by Melatonic on Saturday November 14, 2026 @02:49AM (#50494963)
[delayed]
Re:FAA Issues National Ground Stop by bitbangarang on Friday November 13, 2026 @08:42PM (#50492692)
This is why you don't push PROD on fridays...
Re:FAA Issues National Ground Stop by pugworthy on Friday November 13, 2026 @09:44PM (#50493204)
Mission critical systems that are in use 24/7 are not managed based on week day vs weekend shifts. Or at least shouldn't be. My (hopeful) assumption is that air traffic control isn't based on the "weekends are the off hours" mindset.
Re:FAA Issues National Ground Stop by jmyeet on Friday November 13, 2026 @11:09PM (#50493807)
It's worse than that. It's 3pm on a Friday before Thanksgiving week.

I really hope we get an official or leaked root cause analysis and postmortem of this and it doesn't get buried because all the money in my pockets is willing to bet there was some deep incompetence responsible at the executive level.

Re:FAA Issues National Ground Stop by dalant979 on Friday November 13, 2026 @11:28PM (#50493936)
No it's not. Thanksgiving is on November 26.
Re:FAA Issues National Ground Stop by mnw21cam on Sunday November 15, 2026 @03:54AM (#50505216)
So yes, it is a Friday before thanksgiving week. Two weeks before.
Re:FAA Issues National Ground Stop by axus on Saturday November 14, 2026 @12:59AM (#50494435)
Re:FAA Issues National Ground Stop by stef25 on Saturday November 14, 2026 @03:32AM (#50495206)
Yesterday email went down for half the world's population because MS pushed a faulty update and a day later all US planes are grounded also because of a faulty update, come on what are the chances.

If it was a malicious act, or maybe even two, would they publicly state it was the case ?

Re:FAA Issues National Ground Stop by springtimesun on Saturday November 14, 2026 @04:31AM (#50495495)
If you don’t let juniors ship to prod on Friday afternoons sometimes they will never get their wings.
Re:FAA Issues National Ground Stop by sippingabonedry on Saturday November 14, 2026 @04:35AM (#50495517)
Did anyone question why Microsoft would do an update at 10am on a Thursday?
Re:FAA Issues National Ground Stop by roosterIllusi0n on Saturday November 14, 2026 @02:01PM (#50499760)
It does not matter when they did the updates. What matters now is what the source code is for the code that they deployed. We need to see what the documentation says about human reviews, testing, and certification processes.

If they vibe coded this code, that very likely crosses into criminal charges. That would mean they did not properly vet the code or sign off with human reviewers.

This is the type of corner cutting that caused boeing to kill people when they bypassed development and testing procedures that were required to improve safety.