Why is a single app mapped to multiple routers? Why not just map it to one?
Because you don't want to scale up just by adding more shards that consist of a single router. The distribution of the traffic to each shard isn't random, some applications are more popular than others. Additionally, you don't have any redundancy if a router goes down.
Then you would need another level of HTTP routers that would route to this application specific router. Also the router becomes a single point of failure and possibly a bottleneck: if a backend returns large responses, all of them go through this single router.
Or you could use DNS to map each app to its assigned router. (Of course, when I say "router" I mean at minimum a redundant active/passive pair.)
Heroku likely has their own custom routing software that communicates with their databases to route incoming requests to the correct dynamos, etc.
EDIT: Key problem with the 'at scale' part is that Heroku does not have a single router, they have many many routers. I don't work there, so I have no idea if/why a single HAProxy couldn't handle it all, but obviously it can't or they'd just do that and save money!
Using a HAProxy with 'maxconn' flag works only for that single router instance. If a request arrives at a different router, it has no knowledge that server is wants to route the request to is already processing a request. That knowledge is internal state in a different router.
The key here would be to communicate this state between all routers. As I thought in the OP, the correct way to handle this at scale is some form of multicast - across a 1gbps network, broadcasted status updates should account for a tiny percent of traffic/processing, yet allow intelligent routing that would decrease latency immensely.
So I guess the original question still stands: is there any router project that communicates state between multiple load balancers/routers to allow intelligent routing at scale ?
EDIT2: Here we go - looks like research is being done in this area. Maybe Heroku should read this paper:
http://digitalcommons.unl.edu/cgi/viewcontent.cgi?article=10...