How we handle deploys and failover without disrupting user experience

Re:How we handle deploys and failover without disrupting use... by SpikeGronim on Friday September 28, 2012 @04:38PM (#4587680)

I'll share how wavii does this. We use amazon elb and chef. The first chef recipe that runs on our frontend is "touch /tmp/website_should_drain". The website then returns 404 from the /status health check URL. We wait 30s, update the site, and then rm the draining sentinel file. If the deployment fails the host stays out of the elb. If more than 1/3 host fail we abort the whole deployment. This works well for us and was very simple to implement.

[ Parent | on HN ]

Re:How we handle deploys and failover without disrupting use... by zwily on Friday September 28, 2012 @06:26PM (#4588104)

We've found that when the ELB health check returns non-ok, the ELB will immediately kill any requests in flight to that box. So it's kind of impossible to do a real "drain" with ELB. Are you seeing different behavior?

[ Parent | on HN ]

Re:How we handle deploys and failover without disrupting use... by mattyb on Friday September 28, 2012 @07:04PM (#4588228)

We've seen the same behavior, so we do the draining via HAProxy, which does the right thing.

[ Parent | on HN ]