At least, that's what you do if you use Calico and want to be able to use hypervisor migration when using it with OpenStack.

Just make sure your TOR's can handle the amount of routes necessary, have a default route from the hypervisor to the internet, and from the TOR to the spine, and have the spine advertise a 0's route. So now the spine is the only place where you need beefier routers that can support more than the TOR's in terms of routes.

With some intelligence in the IPAM solution your host will get a /26 (or a /64) and will advertise that entire range, and only a single /32 is advertised if the VM/container moves to another hypervisor host (to support things like live-migration).

That being said, most TOR's can handle quite a large amount of routes these days. When I was at a telco we had some gear that did up to 128k routes, so splitting between IPv4/IPv6 we had 64k routes each. Which is plenty, even for larger clusters.

Yeah, that's what I meant. I thought that was clear from "aggregate an entire rack to a single prefix" but perhaps not.

When I said don't advertise /32s I meant "globally" and as a general rule (I didn't realize what I said would be taken so literally). There can be exceptions, of course, such as in the live-migration case you mentioned.

Easier said than done. Most datacenters are bit more deliberate about allocating addresses and hand them out in non-contiguous CIDRs. The VLAN mentality is still very prevalent. Getting a /20 at a time is pretty common.

Using overlapping IPs puts you right back into the overlay model.

>Assuming everything is nice and hierarchical, you can easily aggregate an entire rack to a single prefix.

Yes, exactly. The trick then becomes how to you ensure that endpoints that get created within the rack get an IP from the prefix? Romana (the project I work on) does this. It lets you capture your network topology for exactly this reason. This is especially important if/when you must filter routes at ToR.