News for hackers. Stuff that matters.
Re:Solving the Mystery of Link Imbalance: A Metastable Failu... by msandford on Wednesday November 26, 2014 @10:36AM (#8662753)
It took two years to figure this out? When I read the first couple of paragraphs my first thought was "the low-latency links are getting dropped from the eligibility pool somehow" and it turns out that was the problem.

I think my intuition there stems from having a lot of full-stack responsibility (so the idea that it's someone else's problem was never a luxury I could afford, since it was always my problem) and having cleaned up a number of other people's very large spaghetti-code disasters.

The other possibility is that I have no special intuition into the problem (this is far more likely) but did have a fresh set of eyes, while all the people who were working on the problem were so intimately familiar with it that they couldn't see the forest for the trees.

Re:Solving the Mystery of Link Imbalance: A Metastable Failu... by ngbronson on Wednesday November 26, 2014 @11:21AM (#8663007)
One of the reasons this took a long time to figure out is that it was a failure amplifier, so there was always a more typical network problem preceding it. Network failures in a data center cause lots of changes to the packets, because of retries, failover, and automatic load balancing, so there were a lot of trees to look at.
Re:Solving the Mystery of Link Imbalance: A Metastable Failu... by msandford on Wednesday November 26, 2014 @10:46PM (#8665899)
That makes a lot more sense. It would have been nice to include some of the troubleshooting process so people can learn from that too. Thanks for sharing!
Re:Solving the Mystery of Link Imbalance: A Metastable Failu... by jerf on Wednesday November 26, 2014 @11:12AM (#8662955)
"It took two years to figure this out? When I read the first couple of paragraphs my first thought was "the low-latency links are getting dropped from the eligibility pool somehow" and it turns out that was the problem."

Well, sure, it's obvious when you're reading a prepared text that is carefully leading you up to that conclusion and has removed all extraneous information. Almost all bugs are shallow once you already know the answer.

Re:Solving the Mystery of Link Imbalance: A Metastable Failu... by msandford on Wednesday November 26, 2014 @12:41PM (#8663489)
The stability of it is the giveaway. It's not a meta-stable failure state, it's a fully stable failure state. There is positive feedback; increased latency on a link begets further link usage; which causes more latency, etc.

Had there been some kind of negative feedback (in the electrical engineering sense of the word) and this still happened somehow it might be a lot more difficult to track down. But introducing negative feedback is what they did to solve the problem, so perhaps it wouldn't have still happened.

http://en.wikipedia.org/wiki/Feedback

Re:Solving the Mystery of Link Imbalance: A Metastable Failu... by ngbronson on Wednesday November 26, 2014 @01:44PM (#8663924)
Metastability comes in when you think about the problem that originally triggered congestion. A normal network overload caused by a bulk transfer, for example, can be fixed by canceling the transfer. Once you add the feedback loop, however, you need to both cancel the transfer and remove lots of other load. I use the term metastable because although it is fully stable (that's the problem!) it isn't the "ground state". Another way to say it is that the metastable states are only local attractors.