News for hackers. Stuff that matters.
Re:FAA Issues National Ground Stop by selcuka on Friday November 13, 2026 @10:07PM (#50493359)
I don't think the hesitation to push to prod Friday afternoons has anything to do with the working schedule of your customers. It's more about whether you'll be able to roll back on a weekend.
Re:FAA Issues National Ground Stop by Geezus_42 on Friday November 13, 2026 @10:19PM (#50493437)
Why wouldn't you be able to rollback on a weekend like any other day?
Re:FAA Issues National Ground Stop by schiffern on Friday November 13, 2026 @11:11PM (#50493825)
Not a family man, I see?
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:28AM (#50496497)
That has nothing to do with it. I don't like working weekends anymore that anyone else. However, every now and then it is necessary for certain systems. Also, if your rollback procedure is go, it isn't issue.
Re:FAA Issues National Ground Stop by schiffern on Saturday November 14, 2026 @09:39AM (#50497372)

  >I don't like working weekend anymore [sic] that [sic] anyone else.
So then why the feigned confusion? It sounds like you already knew the answer to your 'question.'
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @05:19PM (#50501559)
My confusion is people acting like it is blasphemous to deploy on a Friday when I have laid at multiple reasons why it may be necessary.
Re:FAA Issues National Ground Stop by latentsea on Friday November 13, 2026 @11:20PM (#50493887)
You can, but it also means you might have to.
Re:FAA Issues National Ground Stop by ezfe on Friday November 13, 2026 @11:20PM (#50493889)
cause I don't want to work on the weekend. Why would I introduce that risk?
Re:FAA Issues National Ground Stop by kevin_nisbet on Saturday November 14, 2026 @02:13AM (#50494778)
Or if you need more resources, can be a lot harder on a friday evening or weekend to get ahold of someone else to help.
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:21PM (#50498766)
Assuming they don’t have an actual life and aren’t somewhere touching grass.

I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.

Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.

Re:FAA Issues National Ground Stop by dzhiurgis on Saturday November 14, 2026 @02:56AM (#50495002)
It’s ok, claude can do it for you.
Re:FAA Issues National Ground Stop by Rapzid on Saturday November 14, 2026 @05:09AM (#50495692)
Thursday is pretty close to the weekend. Too risky on Thursday?
Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Saturday November 14, 2026 @06:22AM (#50496117)
I'm old, I do prod changes Monday to Wednesday for that reason. I'm not paid to work on weekends. Sure, you can argue that people should be competent enough to never release software with a bug, but I sometimes fuck up.
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:30AM (#50496509)
Or just have a good rollback plan like a snapshot.
Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Saturday November 14, 2026 @08:57AM (#50497065)
If the system I work on needs to be restored from a snapshot then I'll definitely be spending the weekend working. There are systems you can't just rollback and restore a snapshot and all is fine.
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @09:30AM (#50497291)
So when you roll a patch out on a Tuesday morning and it breaks, how much revenue do your client's lose? I'm all for telling a client to eat it because I'm not working on the weekend for some BS, but there are critical systems that CAN'T be offline during the week.
Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Sunday November 15, 2026 @06:26AM (#50506095)
Me and my colleagues don't want to work weekends, so that goes into the planning of deployments. There is always a possibility of failure when releasing software. I don't see the "how much revenue" relevant to what I wrote - if we deploy Friday and it breaks, we need to work weekend.
Re:FAA Issues National Ground Stop by Geezus_42 on Sunday November 15, 2026 @08:02AM (#50506641)
We don't WANT to work on weekends either. That's why it is extraordinarily rare that we do. Plus the fact that most clients have a Dev environment that we test the changes in first.

Clients losing revenue because you broke their application mid-week is important because contributes to your attrition rate. Which, if it's bad enough, will end your company.

Re:FAA Issues National Ground Stop by tuwtuwtuwtuw on Sunday November 15, 2026 @08:56AM (#50506973)
I don't know, it feels like you're arguing based on your knowledge about the system you work on, without taking into account that other systems does not work the same. It's just nonsense. Good luck.
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:33PM (#50498919)
And then there’s the Tower of Hanoi of breaking major changes into tiny pieces that can increment without making rollbacks impossible. Always two more steps than you would think you need and requires patience that management often doesn’t possess.

Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.

If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.

Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.

Re:FAA Issues National Ground Stop by ndsipa_pomu on Saturday November 14, 2026 @10:20AM (#50497726)
Thursday is a good choice, especially if Fridays are less busy than other days as that gives the team a day to work on the issue if necessary and minimises the business impact if it goes bang.
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:26PM (#50498832)
Scrum originally recommended people run sprints from Wednesday to Wednesday so you were doing deployments in the middle of the week. I’ve never worked anywhere that did a Scrum right, and I don’t mean that in the “well then you’re doing it wrong” sense. I used Kanban before Scrum and man is that a downgrade.
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @07:29AM (#50496504)
Because you are patching a critical system that can't be down during the week.
Re:FAA Issues National Ground Stop by ndsipa_pomu on Saturday November 14, 2026 @10:18AM (#50497713)
In which case, why wouldn't you deploy it during the weekend? That way, you're already prepared to be working several hours on a Saturday (presumably earning overtime rates) and so it's no major issue to rollback if it goes wrong.

Friday deployments make no sense if you have teams that work Monday-Friday.

Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:24PM (#50498805)
[delayed]
Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @05:22PM (#50501591)
Why do you only have a single co-worker that knows that error? Why hasn't anyone written shared documentation?
Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @06:19PM (#50502108)
[delayed]
Re:FAA Issues National Ground Stop by Geezus_42 on Sunday November 15, 2026 @07:59AM (#50506621)
We host a private cloud, so the only code we write are occasional shell scripts based on repeatable steps that have already been done manually hundreds or even thousands of times. And yeah, everyone on the team has a pretty thorough knowledge of the applications we host. Documentation is usually enough to fill the blanks.

I'm not saying surprises don't happen, just that they should be extraordinarily rare.

Re:FAA Issues National Ground Stop by hinkley on Sunday November 15, 2026 @05:30PM (#50511556)
I think when we are discussing a very public outage though, we can take it as read that there are any number of problems with the company and this is just the one they got caught out on. And similarly that a lot of replies are from people who have experienced the same with other businesses and/or employers.

As long as you guys don't get underbid by a competitor with a severe case of Dunning Kruger, which I've seen happen too often, then you're good. Usually you have to sell your competence as part of your story. Because you'll never match idiots on price.

Re:FAA Issues National Ground Stop by Geezus_42 on Saturday November 14, 2026 @05:21PM (#50501573)
Friday afternoon is not unusual if you are facing a weekend deployment. Might as well try to get it done Friday afternoon an not have to worry about it the rest of the weekend.
Re:FAA Issues National Ground Stop by DANmode on Friday November 13, 2026 @10:40PM (#50493589)
or want to have to.
Re:FAA Issues National Ground Stop by james_marks on Saturday November 14, 2026 @10:04AM (#50497591)
We avoided Friday deploys because we didn’t have enough usage over the weekend; traffic was oriented on M-F working hours.

It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.

Much better to get fast feedback while the deploy could be monitored against prod traffic.

Re:FAA Issues National Ground Stop by hinkley on Saturday November 14, 2026 @12:18PM (#50498733)
We did Monday morning IST, on a build we declared valid on Friday afternoon - no pressure to declare premature victory there. The cherry on top was sprint demos first thing in the morning PST so when I first joined the project the demos got interrupted on a regular basis.

I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.