Yeah that still holds true to some extent today with the GHA cache. Blacksmith colocates its cache with our CI runners, and ensures that they're in the same local network allowing us to saturate the NIC and provide much faster cache reads/writes. We're also thinking of clever ways to avoid downloading from a cache entirely and instead bind mount cache volumes over the network into the CI runner. Still early days, but stay tuned!

there’s probably a cool consistent hashing solution where jobs are routed to a host that that is likely to have the cache stored locally already and can be mounted into the containers.

Yup! We observed the same thing back before we built Depot. The act of saving/loading cache over a GHA network pretty much negated any performance gain from layer caching. So, we created a solution to persist cache to NVMe disks and orchestrate that across builds so it's immediately available on the next build. All the performance of layer caching without any network transfer.

The registry cache idea is a neat idea, but in practice suffers the same problem.

totally, your approach is the right one and anything reasonable is going to focus on collocating the cache as close as possible to where the build runs.