And that's why you never deploy to prod on a Friday.
"it's computer science, if you make a mistake nobody will die!
Unless you're dealing with nuclear plants, medicals, or airlines"
I spend at least 1/3 of my fridays answering all requests with "Read Only Friday."
I worked for a person who had that mentality-- the "first of the month", "the end of the week", "the start of the quarter", etc. It would be one thing if these dates were tied to legal compliance (tax filings, etc), but that usually wasn't the case.
I'd be sure to time meetings to maximize time for work-- meet late Wednesday because an early Tuesday meeting might get me an "end of the week" deadline, but a late Wednesday meeting might get be "the end of next week". You always had to be aware of the numeric calendar day, too. Gunning for an "end of next week" deadline slot when next Wednesday was the end of the month (or worse, end of the quarter) almost guaranteed you an earlier deadline than you wanted.
The amount of time the work was going to take had no bearing on it.
It was also immensely fun, when setting a deadline in this person's absence, to get a group to agree on a mid-week deadline. You could see it make this person's skin crawl hearing that the work would be done on Tuesday or a Thursday.
So the deployment target seems like it should have been Monday @ 10pm. (Or whenever the bulk of flights have landed, but there's still enough for a validation test)
Overnight periods are when the air travel system largely resets itself from each day's cascading delays, mechanical failures and bad weather.
so you were forced to release updates on weekends? isn't that worse?
Most likely, they HAD to do it.
If something goes down on Saturday there’s a good chance I’m out of cell service
The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.
What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.
Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!
People fought hard for the 40 hour work week and 2 day weekend.
You think that should be given up?
>You think that should be given up?
Not really what I said. My point was that in an era of transition where we started with a 40 hour workweek and a two day weekend, the positive outcome could be a 30 hour workweek and a three day weekend or more. That transition period from one to the other is extremely precarious for everybody and is what we are in currently. It isn't going to transition until a great deal of people make it so, and they only do that when the pain reaches home for them. In the interim, your advice is counterproductive. My take away from this past election: the overwhelming majority are not going to change until it gets real for them. Your advice is to light yourself on fire in the meantime.
I mentioned the fact that I’m often out of cell service on weekends (I like to hike and camp) and so it’s a bad time for an outage.
Are you saying labor conditions need to get worse for everyone in order to get better?
If so I disagree and think we should be advocating for better labor conditions for everyone while protecting things like weekends
>I don't like working weekend anymore [sic] that [sic] anyone else.
So then why the feigned confusion? It sounds like you already knew the answer to your 'question.'I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.
Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.
Clients losing revenue because you broke their application mid-week is important because contributes to your attrition rate. Which, if it's bad enough, will end your company.
Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.
If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.
Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.
Friday deployments make no sense if you have teams that work Monday-Friday.
I'm not saying surprises don't happen, just that they should be extraordinarily rare.
As long as you guys don't get underbid by a competitor with a severe case of Dunning Kruger, which I've seen happen too often, then you're good. Usually you have to sell your competence as part of your story. Because you'll never match idiots on price.
It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.
Much better to get fast feedback while the deploy could be monitored against prod traffic.
I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.
The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.
It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.
I never said things don't go to hell sometimes. That's why you build a good team and train your people.
Many organizations do have round the clock staff but maybe they have the most staff during 9-5 on a particular time zone. So end of day Friday is guaranteeing you'll get a different shift of worker diagnosing failures, i.e. not the author of the deployed change, and possibly fewer available people.
Else, do it during regular business hours and very far from time off (whether that's CoB, weekend, PTO, etc. -- "not on Friday"). If something goes so terribly sideways that users are impacted, at least you have nearly everyone available so hopefully it won't take long to stabilize.
For the FAA it sure as fuck doesn’t. You a bot?
But the entire first months of the Iran War was started on Friday after the markets closed. It took a few weeks for most Americans to catch on.
My money is on some mundane ops issue like "it takes lots of time to get sign off for the upgrade from lots of different stakeholders and the necessary upgrade was falling days or weeks behind, leading to a key IT person to decide to YOLO the upgrade/deployment".
I've heard that Sec Transport Duffy has been pushing FAA upgrades faster than is safe, so maybe the employee rushed it so this upgrade wouldn't hold back some other larger timeline. There have been several articles recently about how the FAA is full of people who are "too risk averse", suggesting that there might be an effort to push riskier projects faster. Having a nationwide outage is one of the likely symptoms of raising risk tolerance.
In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_.
As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.
Yes, this comment is tongue in cheek, but not everyone follows the "don't deploy on Friday" rule, even people in mission critical fields.
Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend.
It’s just dumb.
Are they unionized? Maybe management didn't want to pay time-and-a-half(+) for weekend work?
But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.
Well, it worked so far, I think it is great. 100% availability is a dream.
That’s why for example, you have ATC Zero protocols when communications go out all of a sudden at airports. And I imagine there is something similar for airborne planes that are not near any of their destinations.
It's been pretty cool to see NASA lock in under the new administrator, starr making things in house and not wait on contractors etc.
Foreign cyber-adversaries (the usual suspects ;) are taking notes.
It wouldn't be my pick, that's for sure.
By people with unreliable systems and a lot of myhts behind.
In a well engineered organization, you can deploy whenever you want. Sure, it will not work 100% of the time, but you figure out why, fix the problem and you will be one step closer to the mythical 100%. And so on. Deploying less frequently is what creates more problems.
I really hope we get an official or leaked root cause analysis and postmortem of this and it doesn't get buried because all the money in my pockets is willing to bet there was some deep incompetence responsible at the executive level.
If it was a malicious act, or maybe even two, would they publicly state it was the case ?
If they vibe coded this code, that very likely crosses into criminal charges. That would mean they did not properly vet the code or sign off with human reviewers.
This is the type of corner cutting that caused boeing to kill people when they bypassed development and testing procedures that were required to improve safety.
We are run by morons.
https://www.cnbc.com/2026/08/17/l3harris-ceo-chris-kubasik-l...
You mean he facilitated sale of restricted AI chips to China?
Sorry, but I am tired of the effort to make it look like Jansen's crime was infidelity.
If he just went to whore house he would be fine (well, ignoring a possible expensive divorce).
So from that alone I don't trust the integrity of this source's reporting.
No, they mean "had a sexual relationship with a subordinate".
Which is what the linked article says.
>Sorry, but I am tired of the effort to make it look like Jansen's crime was infidelity.
Strawman. Nobody says that, and nobody cares.
Jensen's "crimes" that we know of are:
1. Having a sexual relationship with a person suspected of being a Chinese spy (who, allegedly, influenced NVIDIA's chip sales to China)
2. Having an undisclosed sexual relationship with a direct report
Both are huge liabilities for the company, as well as national security concerns (someone in a critical position putting themselves into a position for easy blackmail).
None are crimes, all are factors contributing to the firing decision.
>If he just went to whore house he would be fine (well, ignoring a possible expensive divorce).
Same goes for the ousted L3Harris CEO.
Who do you feel is creepy here (or a man, for that matter)?
Infidelity (in itself) is not the thing that got either of those 2 men fired.
That statement doesn't say anything about how anyone here feels about infidelity, nor whether it should or shouldn't be enough of a reason to fire a CEO.
I'm pointing out that looks like there is an effort to kind of turning this scandal as being about infidelity and then in next step conclude that infidelity is not illegal so "Jensen didn't do anything wrong".
The infidelity is bad and typically is connected with character, so somebody who does that is very likely unreliable in other ways as we can see.
Anyway the thing is that talking about infidelity is muddling what actually happened. The US considers this technology to be important for national security and now we are learning that the chips which supposed to be restricted were sold and are in China.
To illustrate the scandal, let's replace the chips with for example F-35 parts (which ironically there was already another scandal a while ago).
poor choice of words
Everything operates normally, as any other day.
I agree that you should do everything possible to make sure there's no downtime. But if there is a time for downtime, it's when there's the least amount of air traffic.
This woodoo like belief you should not deploy on Fridays, Christmas, Valentine’s Day and other magical days is a disservice to the industry.
I feel like people want to go home on Friday and will make sacrifices they would not make on a Wednesday.
Things go wrong all the time, it is not a reason to fear doing stuff.
[1] https://www.l3harris.com/sites/default/files/2021-08/MN_SBSS...
Rollout new router config Thursday night - seems fine at first - issues compound and then remote admin goes down
A bunch of headlines I've seen today indicate that the reason for the circular investments is because NVIDIA, and by extension, TSMC, has gotten in over their heads. NVIDIA ordered massive amounts of wafers from TSMC, and when demand fell due to competition, Jensen tried to backstop companies like OpenAI in order to avoid having to pay for a product with no demand. The interim CEO actually pushed back against him, and managed to negotiate a smaller deal with Open AI, and there is a bunch of stuff surrounding this that media is still uncovering. TSMC has also joined this circular investment stuff. Sorry I don't have links. I saw a few articles on reddit here and elsewhere. Much of the above came from Jensen Huang himself shortly before he resigned. NVIDIA has reserved most of TSMC's advanced nodes, to say nothing of their obligations with Micron, Hynix, etc.
My thoughts after reading various reports? The affair was a cover, not for the smuggling of NVIDIA stuff (although that is an issue), but rather, NVIDIA has been making dangerous bets, the now-CEO called them out. He shut that down, so they went to the board with evidence. I could be wrong, but I bet I'm not. I also bet the smuggling thing was intentional because NVIDIA is completely underwater with wafer allocations, order for RAM, etc.
The reason I bring that stuff up is because it absolutely looks like this bubble is about to pop. This is the part where Dems take over and have to clean up a shit show.
The really fucked up thing? Elon Musk may come out ahead, thanks to him weaseling his way into american retirement plans.
The Internet was purpose designed to tolerate disruption as major as a nuclear war, which it achieves through decentralisation.
Private networks exist to make idiot managers feel more confident while actually dramatically reducing availability by putting all their eggs into a single basket.
That is already well within the era of IPv6, let alone "the Internet".
There's no excuse.
But it does not make it 100% resilient to cascading failures or possible Layer2/3 loops that might happen during a software upgrade, especially in a country wide deployment.
The FAA will give a root cause at some point of why this happenned.
I wouldn’t say anything but this is approximately the only time in my entire life this scene will be relevant.
why a reboot would cause a crash? or it means that an update irreversibly broke the system and it was rebooted but that didn't help?
I learned more from this post in 15sec than I did in 20min from watching the Sec of DHS and Sec of Travel press conference
https://www.wsj.com/us-news/faa-temporarily-halts-flights-ac...
> The outage appears to be affecting a telecommunications network overseen by L3Harris Technologies, according to people briefed on the matter. The network handles crucial radio frequencies that air-traffic controllers use to communicate with pilots, the people said. It wasn’t immediately clear whether the outage also affected the radar feeds that controllers use to keep aircraft safely separated in the skies.
https://www.cbsnews.com/news/faa-issues-nationwide-ground-st...
> An update to the private communications network that connects America's air traffic control system caused the system to reboot, leading to a crash and a communications outage, multiple sources told CBS News. The system is operated by L3Harris, which did not immediately respond to CBS News' request for comment.