Re:Money Trees – Rap Genius Response to Heroku
by mononcqc on Monday February 18, 2013 @11:14AM
(#5238921)
Could you point me to such distributed queue algorithms that can handle failure gracefully? I'm interested in reading on the topic.
Re:Money Trees – Rap Genius Response to Heroku
by bradleyjg on Monday February 18, 2013 @01:01PM
(#5239554)
Re:Money Trees – Rap Genius Response to Heroku
by drusenko on Monday February 18, 2013 @12:23PM
(#5239341)
while not exactly what you're looking for, this might be a good starting point: http://en.wikipedia.org/wiki/Virtual_Router_Redundancy_Proto...
there seems to be some more reading material linked to at the bottom.
Re:Money Trees – Rap Genius Response to Heroku
by jlouis on Monday February 18, 2013 @07:13PM
(#5241728)
I really liked your links to mononcqc :) Because that are way better suggestions than just "intelligent routing" I only skimmed the first one, with Join-Idle-Queueing, but that sounds like a way better algorithm to apply.
Re:Money Trees – Rap Genius Response to Heroku
by bradleyjg on Tuesday February 19, 2013 @01:08PM
(#5245906)
Yeah, Heroku's original method wasn't all that great, and random routing can give surprisingly decent performance in some workloads. Unfortunately the workload they were/are working with doesn't seem to be one of them.
Happily this class of problems is very well studied, and so there is a ton of research out there on various options. I know it can be daunting to write a program directly from a research paper, but hey that's why they make the big bucks.
You are right that there is an inherent latency hit using intelligent routing versus random routing. However in a resource constrained world, where no one can afford infinite dynos, there is also an average latency hit to random routing -- but rather than being largely fixed for all requests it is instead highly variable. While the magnitude of the two need to be factored in, ceteris paribus low variance in latency is better.
As for your single point of failure point, there are distributed queue algorithms that handle router failure gracefully.