News for hackers. Stuff that matters.
Re:Why we don't schedule deployments during off-hours by InclinedPlane on Thursday October 21, 2010 @04:18PM (#1817264)
The better approach is to figure out where the risk in pushing updates is coming from and develop tools to mitigate that. More testing, pseudo-production deployment infrastructure, better change management, more frequent deployments, better understanding of the changes going out with big deployments, etc.

The biggest companies in the world (amazon, google, etc.) who have literally millions of dollars an hour/minute/second at risk if they screw up deployments put in place all of these tools so they can push out deployments at peak times and be confident that they aren't going to destroy their business on accident.

Re:Why we don't schedule deployments during off-hours by scottyallen on Thursday October 21, 2010 @04:47PM (#1817378)
When I was at Google working on web search, we didn't push search infrastructure during peak hours. I think this was mostly due wanting to make sure we had spare serving capacity to failover to, in case the push was broken/had performance problems/crashed/etc. Note that "peak hours" isn't the same as "working hours". Just a few hours when we saw our highest traffic.

However, we also had a strict policy of no pushing in the middle of the night, Fridays, or weekends, for all the reasons the article listed. Most problems on a relatively mature system occur when you're changing stuff. When stuff breaks, you want to have the people involved be fresh and thinking clearly. You also want everyone who might be needed to fix something available, in the office and in front of their computers if something goes wrong. Lastly, it's not sustainable to have your engineers/operations folks regularly giving up their nights and weekends for routine operations. You should save that karma for when everything goes to hell.

This is all predicated on having pushes not negatively affect production traffic when everything goes according to plan, and minimize the impact as much as possible when something does break. One of the best ways I know to do that is to increase the amount of traffic proportional to increases in your confidence that the change is good. The idea is that any push starts off by only affecting a small part of your infrastructure/traffic, and the longer you run it without problems, the more traffic/infrastructure you push it to.

At Shopkick, we do this by pushing first to a single machine, watching it for any errors for a few minutes, then gradually pushing it to the rest of our machines. If anything goes wrong, only a portion of traffic was affected, and the portion is well correlated to the probability of having a problem.

Re:Why we don't schedule deployments during off-hours by InclinedPlane on Thursday October 21, 2010 @04:59PM (#1817412)
Conditional deployment with the ability to control the level of deployment incrementally is a hugely powerful tool. Twitter and amazon both use them, at least for higher risk changes. The ability to "roll back" a deployment completely at the press of a button with a configuration change with no actual running bits changing is very potent.
Re:Why we don't schedule deployments during off-hours by ora600 on Thursday October 21, 2010 @04:44PM (#1817366)
This is a cost-value calculation and neither approach is "best" at all cases. Does the cost of developing the tools and procedures exceed the cost of planned downtimes during the next X years?

Often you build some tools that mitigate some of the risk and plan deployments at a less active hour to mitigate the rest of it.