Secure dynamic overlay networking, plus egress is inherently not a trivial thing. Then add customization/extendability on top.
The important thing to get is that it uses really simple building blocks, very UNIX-y. Every time a new container starts it starts a program (the CNI driver) and pipes in a JSON. And the driver does what it wants (executes iptables, BGP announces a prefix, commits something to its internal DB) and spits out some JSON that tells what IP the container got.
And .. that's it. No magic. Everything has to be configured, arguments passed to some daemon at cluster setup time, and so on. Yes, kubelet and apiserver both have hundreds of command line args, but [almost] all of them are very clearly associated with tuning a functionality (or passing some data forward to some daemon/driver/component) - and thus if you don't touch it but it works, yaay, you don't have to care about it (though of course it might turn out that you'll need to scale up/down/faster/lighter).
The socket API is fine, even with a new shiny API the network fundamentals would have to be managed just the same way. The Linux kernel would be the same, just as Ethernet, IP and so on. Reinventing all of those wheels just to provide transparent pod-to-pod networking doesn't seem to ever worth it.
Kubernetes' networking would arguably have been less complex (multiple CIDR subnets, proxying, etc.) if they'd embraced IPv6. But Google doesn't seem big on IPv6.
// Create an instance representing this service. "my-service" is the
// name of _this_ service. The service should be cleaned up via Close.
svc, _ := connect.NewService("my-service", client)
and when http is used:> The hostname used in the request URL is used to identify the logical service discovery mechanism for the target. It's not actually resolved via DNS but used as a logical identifier for a Consul service discovery mechanism. It has the following specific limitations:
I think it's because they still want to use the socket API, but to "pretend" you can open a socket to a pod, or to a service, etc.
Instead, they should have defined a new API, like "connect('k8s:service:foobar?selectionstrategy=leastloaded')"
Then they could have made simple LD_PRELOAD shims to get their magic API to work with existing software without modification.
There would be no need for millions of IPtables rules, virtual IP addresses, port remapping, proxy processes, etc.