I spend at least 1/3 of my fridays answering all requests with "Read Only Friday."
I worked for a person who had that mentality-- the "first of the month", "the end of the week", "the start of the quarter", etc. It would be one thing if these dates were tied to legal compliance (tax filings, etc), but that usually wasn't the case.
I'd be sure to time meetings to maximize time for work-- meet late Wednesday because an early Tuesday meeting might get me an "end of the week" deadline, but a late Wednesday meeting might get be "the end of next week". You always had to be aware of the numeric calendar day, too. Gunning for an "end of next week" deadline slot when next Wednesday was the end of the month (or worse, end of the quarter) almost guaranteed you an earlier deadline than you wanted.
The amount of time the work was going to take had no bearing on it.
It was also immensely fun, when setting a deadline in this person's absence, to get a group to agree on a mid-week deadline. You could see it make this person's skin crawl hearing that the work would be done on Tuesday or a Thursday.
So the deployment target seems like it should have been Monday @ 10pm. (Or whenever the bulk of flights have landed, but there's still enough for a validation test)
Overnight periods are when the air travel system largely resets itself from each day's cascading delays, mechanical failures and bad weather.
so you were forced to release updates on weekends? isn't that worse?
Most likely, they HAD to do it.
If something goes down on Saturday there’s a good chance I’m out of cell service
The other option is to fail during the workweek. If the software is directly generating money for the users and the developers are judged on what actual business value they provide then failure during business hours is worse.
What seems to be unique in American companies is you’re expected to work shifts or nights but also work m-f 9-5 at the same time, which is really weird.
Same with of call, if you get called at 3am on an on/call rota for a specialised third line escalation many companies still expect you to work the next day, and they might not even pay you for being on call?!
People fought hard for the 40 hour work week and 2 day weekend.
You think that should be given up?
>You think that should be given up?
Not really what I said. My point was that in an era of transition where we started with a 40 hour workweek and a two day weekend, the positive outcome could be a 30 hour workweek and a three day weekend or more. That transition period from one to the other is extremely precarious for everybody and is what we are in currently. It isn't going to transition until a great deal of people make it so, and they only do that when the pain reaches home for them. In the interim, your advice is counterproductive. My take away from this past election: the overwhelming majority are not going to change until it gets real for them. Your advice is to light yourself on fire in the meantime.
I mentioned the fact that I’m often out of cell service on weekends (I like to hike and camp) and so it’s a bad time for an outage.
Are you saying labor conditions need to get worse for everyone in order to get better?
If so I disagree and think we should be advocating for better labor conditions for everyone while protecting things like weekends
>I don't like working weekend anymore [sic] that [sic] anyone else.
So then why the feigned confusion? It sounds like you already knew the answer to your 'question.'I’ve worked with several people who either confessed to or were accused of having taken up pastimes that would guarantee they weren’t available during times they shouldn’t be expected to be available.
Management gets rewarded for getting more work out of us and there’s always someone pushing boundaries trying to leach a little more work out of the staff they have. These same people seem to think burnout is a big mystery, because to admit they caused the problem would be to admit they did something wrong.
Clients losing revenue because you broke their application mid-week is important because contributes to your attrition rate. Which, if it's bad enough, will end your company.
Example, got a feature with a new required field? Add empty field, synthesize a default value for existing rows, start accepting data, validate that data, require the data, use the data.
If you’re doing deployments once a week, you end up juggling two stories so you’re not sitting on your hands the whole time, and two stories divided your attention, so the odds of fucking it up go up anyway even though you’re being careful on paper.
Meanwhile one of your coworkers is doing the same dance on another feature, so you start tripping over each other as team size increases.
Friday deployments make no sense if you have teams that work Monday-Friday.
I'm not saying surprises don't happen, just that they should be extraordinarily rare.
As long as you guys don't get underbid by a competitor with a severe case of Dunning Kruger, which I've seen happen too often, then you're good. Usually you have to sell your competence as part of your story. Because you'll never match idiots on price.
It meant that issues would sit undiscovered over the weekend until Monday morning, and lead to a week starting in chaos.
Much better to get fast feedback while the deploy could be monitored against prod traffic.
I don’t think most people fully appreciated how much work a handful of us did to make sign off faster and deployment less error prone.
The reason you don’t want to push on a Friday, or overnight, or weekend, is your experts and the top bosses tend to work m-f 9-5, so pushing early in the day and early in the week ensures more people are available to fix problems and run comms.
It’s easy to think things aren’t going to go to hell, but look at the huge companies that screw things up. Amazon. GitHub. It could happen to you, too.
I never said things don't go to hell sometimes. That's why you build a good team and train your people.
Many organizations do have round the clock staff but maybe they have the most staff during 9-5 on a particular time zone. So end of day Friday is guaranteeing you'll get a different shift of worker diagnosing failures, i.e. not the author of the deployed change, and possibly fewer available people.
Else, do it during regular business hours and very far from time off (whether that's CoB, weekend, PTO, etc. -- "not on Friday"). If something goes so terribly sideways that users are impacted, at least you have nearly everyone available so hopefully it won't take long to stabilize.
For the FAA it sure as fuck doesn’t. You a bot?
But the entire first months of the Iran War was started on Friday after the markets closed. It took a few weeks for most Americans to catch on.
My money is on some mundane ops issue like "it takes lots of time to get sign off for the upgrade from lots of different stakeholders and the necessary upgrade was falling days or weeks behind, leading to a key IT person to decide to YOLO the upgrade/deployment".
I've heard that Sec Transport Duffy has been pushing FAA upgrades faster than is safe, so maybe the employee rushed it so this upgrade wouldn't hold back some other larger timeline. There have been several articles recently about how the FAA is full of people who are "too risk averse", suggesting that there might be an effort to push riskier projects faster. Having a nationwide outage is one of the likely symptoms of raising risk tolerance.
In what sense? It was well reported at the time, so presumably most people know about it. I'd be interested in what makes you think the time and day of week chosen would delay market or public reaction be _weeks_.
As an insider to the technology industry, I would surmise the technical reason for it is that they didn't want to work over the weekend.
Yes, this comment is tongue in cheek, but not everyone follows the "don't deploy on Friday" rule, even people in mission critical fields.
Isn't that exactly what they did? If you do major change on Friday at 3pm you likely will work over weekend.
It’s just dumb.
Are they unionized? Maybe management didn't want to pay time-and-a-half(+) for weekend work?
But what I’m hearing is that one company “L3Harris” could take down the airfristructure anytime it wanted to. That’s not great.
Well, it worked so far, I think it is great. 100% availability is a dream.
That’s why for example, you have ATC Zero protocols when communications go out all of a sudden at airports. And I imagine there is something similar for airborne planes that are not near any of their destinations.
It's been pretty cool to see NASA lock in under the new administrator, starr making things in house and not wait on contractors etc.
Foreign cyber-adversaries (the usual suspects ;) are taking notes.
It wouldn't be my pick, that's for sure.
By people with unreliable systems and a lot of myhts behind.
In a well engineered organization, you can deploy whenever you want. Sure, it will not work 100% of the time, but you figure out why, fix the problem and you will be one step closer to the mythical 100%. And so on. Deploying less frequently is what creates more problems.
I really hope we get an official or leaked root cause analysis and postmortem of this and it doesn't get buried because all the money in my pockets is willing to bet there was some deep incompetence responsible at the executive level.
If it was a malicious act, or maybe even two, would they publicly state it was the case ?
If they vibe coded this code, that very likely crosses into criminal charges. That would mean they did not properly vet the code or sign off with human reviewers.
This is the type of corner cutting that caused boeing to kill people when they bypassed development and testing procedures that were required to improve safety.