News for hackers. Stuff that matters.

A comment on Downtime last Saturday. Parent comment

Re:Downtime last Saturday by kyrra on Wednesday December 26, 2012 @04:44PM (#4971073)
Failover is meant to deal with hardware failures, which will tend to work just fine. But if the node you are failing over onto was already has 60% capacity and you add another 60% capacity during the failover, things are going to get worse.

The top-level systems probably need to be able to deal with increased latency or timeouts, and properly handle retries and throttling of traffic.

If you have some HA failover setup going but your alternate is already being used more for load balancing than for failover, problems like this will occur.

(I used to work on failover drivers for a SAN).

[ Parent | on HN ]
Re:Downtime last Saturday by jackowayed on Wednesday December 26, 2012 @06:36PM (#4971539)
GitHub's failover problems have never been load-related. GitHub has pairs of fileservers where one is the master and the other's sole job is to follow along with the master and take over if it thinks the master is down, so when they do failover, it is to a node with just as much capacity as the previous master.

All the failover problems I can think of since they moved to this architecture 4 years ago have been coordination problems where something undesired happened when transitioning from one member of a pair to another. In this case, network problems lead them to a state where both members of a pair thought they were the master.

[ Parent | on HN ]