News for hackers. Stuff that matters.
Re:Kubernetes Networking: Behind the Scenes by londons_explore on Wednesday October 16, 2019 @12:15PM (#21272046)
I feel like k8s networking is 10x more complex than it needs to be...

I think it's because they still want to use the socket API, but to "pretend" you can open a socket to a pod, or to a service, etc.

Instead, they should have defined a new API, like "connect('k8s:service:foobar?selectionstrategy=leastloaded')"

Then they could have made simple LD_PRELOAD shims to get their magic API to work with existing software without modification.

There would be no need for millions of IPtables rules, virtual IP addresses, port remapping, proxy processes, etc.

Re:Kubernetes Networking: Behind the Scenes by pas on Wednesday October 16, 2019 @06:06PM (#21276461)
It's complex because K8S is used in a wide variety of complex environments.

Secure dynamic overlay networking, plus egress is inherently not a trivial thing. Then add customization/extendability on top.

The important thing to get is that it uses really simple building blocks, very UNIX-y. Every time a new container starts it starts a program (the CNI driver) and pipes in a JSON. And the driver does what it wants (executes iptables, BGP announces a prefix, commits something to its internal DB) and spits out some JSON that tells what IP the container got.

And .. that's it. No magic. Everything has to be configured, arguments passed to some daemon at cluster setup time, and so on. Yes, kubelet and apiserver both have hundreds of command line args, but [almost] all of them are very clearly associated with tuning a functionality (or passing some data forward to some daemon/driver/component) - and thus if you don't touch it but it works, yaay, you don't have to care about it (though of course it might turn out that you'll need to scale up/down/faster/lighter).

The socket API is fine, even with a new shiny API the network fundamentals would have to be managed just the same way. The Linux kernel would be the same, just as Ethernet, IP and so on. Reinventing all of those wheels just to provide transparent pod-to-pod networking doesn't seem to ever worth it.

Re:Kubernetes Networking: Behind the Scenes by atombender on Wednesday October 16, 2019 @01:52PM (#21273228)
How is that different from today's use of KubeDNS?

Kubernetes' networking would arguably have been less complex (multiple CIDR subnets, proxying, etc.) if they'd embraced IPv6. But Google doesn't seem big on IPv6.

Re:Kubernetes Networking: Behind the Scenes by 0xEFF on Wednesday October 16, 2019 @02:21PM (#21273615)
API incompatibility is largely why AppEngine failed. Cloud adoption suffers greatly when the platform is not compatible with the data center.
Re:Kubernetes Networking: Behind the Scenes by justinsaccount on Wednesday October 16, 2019 @12:42PM (#21272401)
That is basically how consul connect works: https://www.consul.io/docs/connect/native/go.html
Re:Kubernetes Networking: Behind the Scenes by atombender on Wednesday October 16, 2019 @01:48PM (#21273176)
Not as I understand it. It uses DNS resolution, so you connect to something like foo.web.service.consul. Kubernetes's KubeDNS works exactly the same way with Kubernetes service objects.
Re:Kubernetes Networking: Behind the Scenes by justinsaccount on Wednesday October 16, 2019 @03:09PM (#21274187)
From that page:

  // Create an instance representing this service. "my-service" is the
  // name of _this_ service. The service should be cleaned up via Close.
  svc, _ := connect.NewService("my-service", client)
and when http is used:

> The hostname used in the request URL is used to identify the logical service discovery mechanism for the target. It's not actually resolved via DNS but used as a logical identifier for a Consul service discovery mechanism. It has the following specific limitations:

Re:Kubernetes Networking: Behind the Scenes by atombender on Wednesday October 16, 2019 @03:32PM (#21274495)
Ah, I see. That's presumably possible because Go has its own DNS resolver, and it (net.DefaultResolver) can be overridden by libraries; no LD_PRELOAD needed.
Re:Kubernetes Networking: Behind the Scenes by justinsaccount on Wednesday October 16, 2019 @04:06PM (#21274887)
it's because connect.NewService is the consul client, not something like net.DialTCP
Re:Kubernetes Networking: Behind the Scenes by atombender on Wednesday October 16, 2019 @04:25PM (#21275128)
Oh, they require all TCP connections to go through that. That's a bit heavy-handed — requires rewriting all client code to adapt to that pattern. The nice thing about KubeDNS is that you don't have to. Same with Istio.
Re:Kubernetes Networking: Behind the Scenes by justinsaccount on Wednesday October 16, 2019 @05:29PM (#21275997)
I brought up the native client because you mentioned "they should have defined a new API".
Re:Kubernetes Networking: Behind the Scenes by atombender on Wednesday October 16, 2019 @05:33PM (#21276036)
That wasn't me. I'd argue the opposite. Though I'd also maybe argue that SRV records is what we should have standardized on for this stuff a long time ago.
Re:Kubernetes Networking: Behind the Scenes by peterwwillis on Wednesday October 16, 2019 @05:29PM (#21275989)
Integrating with app code is optional. The default is to use a sidecar.