This is a config value; if you want more containers per node, use a /23 or a /22 instead. It's entirely up to the operator, there's nothing magical about the default choice of /24 (except for it being easier to perform arithmetic on).
> Thirdly it ties up a huge address ranges with _no_ flexibility.
If you're using 10/8, then you have 16 bits' worth of /24 subnets, so 65k nodes by default. It's true that there are some companies in the world that have to worry about this limit, but for almost everybody I don't think this is a real problem.
Indeed, but its something extra that _you_ have think about after you've setup your VPC (if youre on AWS) not only does it mean you can deploy/configure un routable IPs by accident, its using a mechanism that _slows down_ your VPC, and adds a minefield of confinguration errors. its just madness.
It's just a LAN, why would you ever statically assign IPs? especally at scale, especially if you have a dynamic ever changing workload. Deploy a pod, two network interfaces, macvlan & AWS does the rest. Put a cloudwatch alert for DHCP exhaustion, or put a resource limit in for each AZ.
Put it this way: Why do you want to have to think about subnets _after_ you've created your VPC? (unless you've reached a limit...)
But this is the problem that most designs are trying to solve. Large L2s are notoriously fragile. 1,000 nodes, 50-100 pods/node is a lot of ARPs. And sometimes you want partitions between endpoints for security/isolation.
I agree with you about static assignment of addresses. But that's why (most) CNIs work with a controller of some kind for IPAM.
IMO, the problem complexity is hard to compress. You need to distribute/manage MAC addresses, routes, and/or state. Different designs would favor one over another.
In this case I think the traditional model works well, has excellent documentation, and scale much better than the alternatives, especailly in AWS.
Firstly _statically_ assigning an address range to each node is utter madness, firstly it limits the containers you can have. Secondly its terribly inflexible, its perfectly possible to have a beefy server have more than 254 containers running.
Thirdly it ties up a huge address ranges with _no_ flexibility. If you have nodes assigned to certain duties (like DB pods) then it can only realistically have a few containers. So the rest of the address range is wasted.
What is so frustrating is that all of this is automatically taken care of using DHCP and macvlan.
In the example thats linked, why isn't there a second adaptor on a different VLAN? Thats a far more simple and visible way of linking things together. I just don't see why you'd want to willingly fiddle with routing table when on a normal flat network its done for you, automatically.