You're correct on HAProxy - I didn't realize HAProxy had a 'maxconn' flag. I'm guessing the issue for Heroku is that that don't have 1 router/haproxy per app, and that 1 server can handle many dynamos/apps.

Heroku likely has their own custom routing software that communicates with their databases to route incoming requests to the correct dynamos, etc.

EDIT: Key problem with the 'at scale' part is that Heroku does not have a single router, they have many many routers. I don't work there, so I have no idea if/why a single HAProxy couldn't handle it all, but obviously it can't or they'd just do that and save money!

Using a HAProxy with 'maxconn' flag works only for that single router instance. If a request arrives at a different router, it has no knowledge that server is wants to route the request to is already processing a request. That knowledge is internal state in a different router.

The key here would be to communicate this state between all routers. As I thought in the OP, the correct way to handle this at scale is some form of multicast - across a 1gbps network, broadcasted status updates should account for a tiny percent of traffic/processing, yet allow intelligent routing that would decrease latency immensely.

So I guess the original question still stands: is there any router project that communicates state between multiple load balancers/routers to allow intelligent routing at scale ?

EDIT2: Here we go - looks like research is being done in this area. Maybe Heroku should read this paper:

http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=10...

Why is a single app mapped to multiple routers? Why not just map it to one?

Because you don't want to scale up just by adding more shards that consist of a single router. The distribution of the traffic to each shard isn't random, some applications are more popular than others. Additionally, you don't have any redundancy if a router goes down.

Then you would need another level of HTTP routers that would route to this application specific router. Also the router becomes a single point of failure and possibly a bottleneck: if a backend returns large responses, all of them go through this single router.

Or you could use DNS to map each app to its assigned router. (Of course, when I say "router" I mean at minimum a redundant active/passive pair.)

What happens if your single router catches on fire?

"Is there any Heroku app larger than that?"

Yes. Consider that Heroku publishes DNS records for all their customers to the same set of nodes; those nodes must be responsible for routing all Heroku traffic. Sharding customers onto individual Haproxy LBs would require either updating an upstream load balancer (which just pushes the problem around) or updating DNS (which comes with long convergence times).