News for hackers. Stuff that matters.
Re:Why we don't schedule deployments during off-hours by Murkin on Thursday October 21, 2010 @02:08PM (#1816717)
Our deployment process was always at night (10,000+ user org).

Even if your process is great and there is no downtime during upgrades, any problem might cause the whole organization to loose thousands of work hours (instead of just your team working during the night).

If your deploying a "nice-to-have" app, its all fine.

Otherwise, get to work late, order a pizza and prepare for a night of adventure.

Re:Why we don't schedule deployments during off-hours by InclinedPlane on Thursday October 21, 2010 @04:18PM (#1817264)
The better approach is to figure out where the risk in pushing updates is coming from and develop tools to mitigate that. More testing, pseudo-production deployment infrastructure, better change management, more frequent deployments, better understanding of the changes going out with big deployments, etc.

The biggest companies in the world (amazon, google, etc.) who have literally millions of dollars an hour/minute/second at risk if they screw up deployments put in place all of these tools so they can push out deployments at peak times and be confident that they aren't going to destroy their business on accident.

Re:Why we don't schedule deployments during off-hours by scottyallen on Thursday October 21, 2010 @04:47PM (#1817378)
When I was at Google working on web search, we didn't push search infrastructure during peak hours. I think this was mostly due wanting to make sure we had spare serving capacity to failover to, in case the push was broken/had performance problems/crashed/etc. Note that "peak hours" isn't the same as "working hours". Just a few hours when we saw our highest traffic.

However, we also had a strict policy of no pushing in the middle of the night, Fridays, or weekends, for all the reasons the article listed. Most problems on a relatively mature system occur when you're changing stuff. When stuff breaks, you want to have the people involved be fresh and thinking clearly. You also want everyone who might be needed to fix something available, in the office and in front of their computers if something goes wrong. Lastly, it's not sustainable to have your engineers/operations folks regularly giving up their nights and weekends for routine operations. You should save that karma for when everything goes to hell.

This is all predicated on having pushes not negatively affect production traffic when everything goes according to plan, and minimize the impact as much as possible when something does break. One of the best ways I know to do that is to increase the amount of traffic proportional to increases in your confidence that the change is good. The idea is that any push starts off by only affecting a small part of your infrastructure/traffic, and the longer you run it without problems, the more traffic/infrastructure you push it to.

At Shopkick, we do this by pushing first to a single machine, watching it for any errors for a few minutes, then gradually pushing it to the rest of our machines. If anything goes wrong, only a portion of traffic was affected, and the portion is well correlated to the probability of having a problem.

Re:Why we don't schedule deployments during off-hours by InclinedPlane on Thursday October 21, 2010 @04:59PM (#1817412)
Conditional deployment with the ability to control the level of deployment incrementally is a hugely powerful tool. Twitter and amazon both use them, at least for higher risk changes. The ability to "roll back" a deployment completely at the press of a button with a configuration change with no actual running bits changing is very potent.
Re:Why we don't schedule deployments during off-hours by ora600 on Thursday October 21, 2010 @04:44PM (#1817366)
This is a cost-value calculation and neither approach is "best" at all cases. Does the cost of developing the tools and procedures exceed the cost of planned downtimes during the next X years?

Often you build some tools that mitigate some of the risk and plan deployments at a less active hour to mitigate the rest of it.

Re:Why we don't schedule deployments during off-hours by danilocampos on Thursday October 21, 2010 @02:40PM (#1816861)
> prepare to [have] a night of adventure.

This really put a smile on my face.

That's the best approach to take with anything, but it's especially applicable to technical stuff that's complex and hard to predict.

I've wasted a fair amount of time being pissy and figured out this: with computers, as in life, the only thing over which you have complete control is your attitude. Plan to have fun and learn things and even the sub-optimal can be a good time.

Re:Why we don't schedule deployments during off-hours by itsderek23 on Thursday October 21, 2010 @03:07PM (#1816986)
> prepare to [have] a night of adventure.

This is exactly why we don't deploy at night. In what other line of work do you schedule complex work when you're going to be tired?

Re:Why we don't schedule deployments during off-hours by Murkin on Thursday October 21, 2010 @03:12PM (#1817008)
I really hope they don't start fixing roads during the day.. you know.. so the workers are not tired.

Or major electrical upgrades.. or phones.. or local internet backbone.. or water.. or what ever other utility that is more important to my life than github (which was down today)

How hard is it to work at night one time ??

Re:Why we don't schedule deployments during off-hours by LaGrange on Thursday October 21, 2010 @06:58PM (#1817793)
The big point seems to be "Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment."

It is like a road stuck in a permanent rush hour -- if the time of repair has little effect on the users annoyance, then it's a good idea to pick the time to maximize your performance.

Working at night isn't hard in the meaning "I don't know if I can pull that off," but in the "I don't know how well will I pull that off" meaning. Anyone can pull an all-nighter or two, but most people will make more errors in the night than during the day.

Re:Why we don't schedule deployments during off-hours by InclinedPlane on Thursday October 21, 2010 @04:20PM (#1817273)
For roads it's almost impossible to avoid impacting use during maintenance, the same isn't, or shouldn't, be true with software.
Re:Why we don't schedule deployments during off-hours by maw on Thursday October 21, 2010 @06:54PM (#1817784)
1) It takes a tough man to make a tender chicken.

2) You should know what you're going to do ahead of time, and you should have done tests beforehand. The after-hours work isn't that complex if you're doing it right.

Re:Why we don't schedule deployments during off-hours by lsc on Friday October 22, 2010 @11:42AM (#1820059)
1) It takes a tough man to make a tender chicken. 2) You should know what you're going to do ahead of time, and you should have done tests beforehand. The after-hours work isn't that complex if you're doing it right.

1. you have no idea how many systems have been utterly destroyed by "tough men" working beyond their limits. I have seen this many places I've worked as a SysAdmin... it's like a mind disease, and it's not any good for anyone.

2. You need to prepare ahead of time, and you need an easy way to roll back. but you /always/ need to be prepared just in case something goes wrong.

This is the other classic mistake: Make the new guy stay up all night handling the upgrade. I mean, the senior guys have tested it, right? It ends in tears for all involved. the new guy is the least able to deal with the stresses of doing the upgrade (and the rollback, and even if you test the upgrade, nobody tests the rollback)

Do your upgrades when your senior people are around, preferably towards the beginning of their shift. If everything goes as planned, it won't matter when you do it, because downtime will be minimal either way. If, however, it throws up on your shoes, I don't know about you, but I am significantly more effective at dealing with problems when I'm fresh. And yeah, that means I'm a wimp. But when you get down to it, most of us are.

Re:Why we don't schedule deployments during off-hours by carbocation on Thursday October 21, 2010 @03:20PM (#1817039)
Medicine? (Although your hand is usually forced into doing night procedures--it's not so much a choice, I agree.)
Re:Why we don't schedule deployments during off-hours by mikedanko on Friday October 22, 2010 @12:22AM (#1818567)
I missed some context there. I thought "yes, I need medicine after night work". Why did I completely miss the mark? Swing shifts all week.
Re:Why we don't schedule deployments during off-hours by carbocation on Friday October 22, 2010 @09:05AM (#1819512)
Heh, I meant the field of medicine. But I can understand your perspective :)
Re:Why we don't schedule deployments during off-hours by opiuygtfrtghyju on Thursday October 21, 2010 @04:10PM (#1817230)
The trouble with that is the deployment is expected to be simple and under your control.

The real fun bit comes when users get at it. By doing the deployment at night you find out if it worked at 9:00am the next morning when 1000s of employees connect try and do some work. If there are any issues you are then in deep xxxx

By doing deployment in the afternoon, people are offline for the hour or so it takes to deploy but then you have all night to fix any issues.

Re:Why we don't schedule deployments during off-hours by scottyallen on Thursday October 21, 2010 @04:52PM (#1817396)
Deploying new code should never be an "adventure". It should be a simple, replicable process that allows you to safely stop or rollback at any given time. It should have enough checks in place that the impact from any given mistake is minimized. There should never be any "oh shit" moments.
Re:Why we don't schedule deployments during off-hours by cullenking on Thursday October 21, 2010 @06:51PM (#1817776)
There will always be "oh shit" moments, no matter how much you think there won't. Yeah, you can mitigate alot of them, but things get through. Even using deploys which can be rolled back, frozen dependencies or bundler for us rails folk, shit can and will happen.
Re:Why we don't schedule deployments during off-hours by scottyallen on Thursday October 21, 2010 @08:46PM (#1818120)
You're right - I should be more clear what I mean. "Oh shit" moments caused by a change you made due to pushing a new piece of code should never last more than a minute or two, between the time you realize something went wrong and the time you run the command that reverts things back to the known good state. At Shopkick, this usually means running a command which flips a symlink that points at the current code/config to use and restarts/reloads the server in question.
Re:Why we don't schedule deployments during off-hours by rmc00 on Friday October 22, 2010 @07:41PM (#1821967)
I work for a company that provides a mission critical service to a lot of people. Maybe we're just spoiled, but we have multiple failover environments. That means that we can deploy in the afternoon by failing the service over to a hot backup during the deployment. When the deploy is finished, we fail the backup system to the production environment and we deploy to the backups while the primary is running. Since there's 0 downtime, we like to work through the process in the morning/afternoons. No all-nighters here, and its definitely not a nice-to-have service. It's all about architecture and design.
Re:Why we don't schedule deployments during off-hours by masklinn on Thursday October 21, 2010 @03:43PM (#1817125)
> Our deployment process was always at night (10,000+ user org).

You didn't even get to the second paragraph of the article? Really?

> Many web applications, including Scout, have customers around the world. There isn’t a perfect time for a deployment.

Re:Why we don't schedule deployments during off-hours by sparky on Thursday October 21, 2010 @04:20PM (#1817276)
FWIW, many 10k+ user organizations are "around the world" as well.

Web apps for server cluster admins have customer bases evenly distributed across every time zone on the planet? Really?

There may not be a perfect time, but there are almost certainly better and worse times.

Re:Why we don't schedule deployments during off-hours by LaGrange on Thursday October 21, 2010 @07:01PM (#1817800)
Maybe the differences are not big enough to justify increasing the probability of mistakes.
Re:Why we don't schedule deployments during off-hours by sparky on Thursday October 21, 2010 @10:03PM (#1818302)
Granted. This is a much more reasonable response to the top of this thread than sanctimonious quoting.

See also scottyallen's comment here (http://news.ycombinator.com/item?id=1817378). If Google web search has (or recently had) such a thing as "peak hours," I'll hazard a guess that <your web app> does too.