For what we're doing, we actually need to consider more than just load. Since our LBs are distributed globally, we also want to make sure we're sending requests to backends that are geographically near them.
We can do this by tracking latency between the load balancer and origin servers, then using it to restrict the candidate pool we're going to choose two from at random.
Consistent hashing is used to always attach a request to the same host. It's the opposite of load balancing.
Load balancing algorithms (least connection, business, etc...) are used to distribute requests across servers as well as possible to maximize performances.
You want to minimize load on all servers, but you also want to pack things up efficiently (so minimize operational costs), but of course you want the benefits of caching, so you want requests from a sessions to land on the same node/server/box.
Basically a multi-dimensional optimization problem. Completely solvable with constraints. Let the business people decide what's more important, latency or throughput or low cost of operations.
Problem with consistent hashing:
Problem with Power of 2 Load Balancing: Instead, he used something called Consistent Hashing with Bounded Loads.[1] https://medium.com/vimeo-engineering-blog/improving-load-bal...