The biggest companies in the world (amazon, google, etc.) who have literally millions of dollars an hour/minute/second at risk if they screw up deployments put in place all of these tools so they can push out deployments at peak times and be confident that they aren't going to destroy their business on accident.
However, we also had a strict policy of no pushing in the middle of the night, Fridays, or weekends, for all the reasons the article listed. Most problems on a relatively mature system occur when you're changing stuff. When stuff breaks, you want to have the people involved be fresh and thinking clearly. You also want everyone who might be needed to fix something available, in the office and in front of their computers if something goes wrong. Lastly, it's not sustainable to have your engineers/operations folks regularly giving up their nights and weekends for routine operations. You should save that karma for when everything goes to hell.
This is all predicated on having pushes not negatively affect production traffic when everything goes according to plan, and minimize the impact as much as possible when something does break. One of the best ways I know to do that is to increase the amount of traffic proportional to increases in your confidence that the change is good. The idea is that any push starts off by only affecting a small part of your infrastructure/traffic, and the longer you run it without problems, the more traffic/infrastructure you push it to.
At Shopkick, we do this by pushing first to a single machine, watching it for any errors for a few minutes, then gradually pushing it to the rest of our machines. If anything goes wrong, only a portion of traffic was affected, and the portion is well correlated to the probability of having a problem.
Often you build some tools that mitigate some of the risk and plan deployments at a less active hour to mitigate the rest of it.
This really put a smile on my face.
That's the best approach to take with anything, but it's especially applicable to technical stuff that's complex and hard to predict.
I've wasted a fair amount of time being pissy and figured out this: with computers, as in life, the only thing over which you have complete control is your attitude. Plan to have fun and learn things and even the sub-optimal can be a good time.
This is exactly why we don't deploy at night. In what other line of work do you schedule complex work when you're going to be tired?
Or major electrical upgrades.. or phones.. or local internet backbone.. or water.. or what ever other utility that is more important to my life than github (which was down today)
How hard is it to work at night one time ??
It is like a road stuck in a permanent rush hour -- if the time of repair has little effect on the users annoyance, then it's a good idea to pick the time to maximize your performance.
Working at night isn't hard in the meaning "I don't know if I can pull that off," but in the "I don't know how well will I pull that off" meaning. Anyone can pull an all-nighter or two, but most people will make more errors in the night than during the day.
2) You should know what you're going to do ahead of time, and you should have done tests beforehand. The after-hours work isn't that complex if you're doing it right.
1. you have no idea how many systems have been utterly destroyed by "tough men" working beyond their limits. I have seen this many places I've worked as a SysAdmin... it's like a mind disease, and it's not any good for anyone.
2. You need to prepare ahead of time, and you need an easy way to roll back. but you /always/ need to be prepared just in case something goes wrong.
This is the other classic mistake: Make the new guy stay up all night handling the upgrade. I mean, the senior guys have tested it, right? It ends in tears for all involved. the new guy is the least able to deal with the stresses of doing the upgrade (and the rollback, and even if you test the upgrade, nobody tests the rollback)
Do your upgrades when your senior people are around, preferably towards the beginning of their shift. If everything goes as planned, it won't matter when you do it, because downtime will be minimal either way. If, however, it throws up on your shoes, I don't know about you, but I am significantly more effective at dealing with problems when I'm fresh. And yeah, that means I'm a wimp. But when you get down to it, most of us are.
The real fun bit comes when users get at it. By doing the deployment at night you find out if it worked at 9:00am the next morning when 1000s of employees connect try and do some work. If there are any issues you are then in deep xxxx
By doing deployment in the afternoon, people are offline for the hour or so it takes to deploy but then you have all night to fix any issues.
You didn't even get to the second paragraph of the article? Really?
> Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment.
Web apps for server cluster admins have customer bases evenly distributed across every time zone on the planet? Really?
There may not be a perfect time, but there are almost certainly better and worse times.
See also scottyallen's comment here (http://news.ycombinator.com/item?id=1817378). If Google web search has (or recently had) such a thing as "peak hours," I'll hazard a guess that <your web app> does too.
Even if your process is great and there is no downtime during upgrades, any problem might cause the whole organization to loose thousands of work hours (instead of just your team working during the night).
If your deploying a "nice-to-have" app, its all fine.
Otherwise, get to work late, order a pizza and prepare for a night of adventure.