News for hackers. Stuff that matters.
Re:Kubernetes is our generation's Multics by dekhn on Wednesday July 21, 2021 @02:57PM (#27910515)
I have borg experience and my experience with k8s was extremely negative. Most of my time was spent diagnosing self-inflicted probmems by the k8s framework.

I've been trying nomad lately and it's a bit more direct.

Re:Kubernetes is our generation's Multics by jedberg on Wednesday July 21, 2021 @03:03PM (#27910585)
I think that's because Borg comes with a team of engineers who keep it running and make it easy.

I've had a similar experience with Cassandra. Using Cassandra at Netflix was a joy because it always just worked. But there was also a team of engineers who made sure that was the case. Running it elsewhere was always fraught with peril.

Re:Kubernetes is our generation's Multics by dekhn on Wednesday July 21, 2021 @03:12PM (#27910696)
yes several of the big benefits are: the people who run borg (and the ecosystem) are well run (for the most part). And, the ability to find them in chat and get them to fix things for you (or explain some sharp edge).
Re:Kubernetes is our generation's Multics by jrockway on Wednesday July 21, 2021 @03:31PM (#27910971)
I have borg experience and I think Kubernetes is great. Before borg, I would basically never touch production -- I would let someone else handle all that because it was always a pain. When I left Google, I had to start releasing software (because every other developer is also in that "let someone else handle it" mindset), and Kubernetes removed a lot of the pain. Write a manifest. Change the version. Apply. Your new shit is running. If it crashes, traffic is still directed to the working replicas. Everyone on my team can release their code to any environment with a single click. Nobody has ever ssh'd to production. It just works.

I do understand people's complaints, however.

Setting up "the rest" of the system involves making a lot of decisions. Observability requires application support, and you have to set up the infrastructure yourself. People generally aren't willing to do that, and so are upset when their favorite application doesn't work their favorite observability stack. (I remember being upset that my traces didn't propagate from Envoy to Grafana, because Envoy uses the Zipkin propagation protocol and Grafana uses Jaeger. However, Grafana is open source and I just added that feature. Took about 15 minutes and they released it a few days later, so... the option is available to people that demand perfection.)

Auth is another issue that has been punted on. Maybe your cloud provider has something. Maybe you bought something. Maybe the app you want to run supports OIDC. To me, the dream of the container world is that applications don't have to focus on these things -- there is just persistent authentication intrinsic to the environment, and your app can collect signals and make a decision if absolutely necessary. But that's not the way it worked out -- BeyondCorp style authentication proxies lost to OIDC. So if you write an application, your team will be spending the first month wiring that in, and the second month documenting all the quirks with Okta, Auth0, Google, Github, Gitlab, Bitbucket, and whatever other OIDC upstreams exist. Big disaster. (I wrote https://github.com/jrockway/jsso2 and so this isn't a problem for me personally. I can run any service I want in my Kubernetes cluster, and authenticate to it with my FaceID on my phone, or a touch of my Yubikey on my desktop. Applications that want my identity can read the signed header with extra information and verify it against a public key. But, self-hosting auth is not a moneymaking business, so OIDC is here to stay, wasting thousands of hours of software engineering time a day.)

Ingress is the worst of Kubernetes' APIs. My customers run into Ingress problems every day, because we use gRPC and keeping HTTP/2 streams intact from client to backend is not something it handles well. I have completely written it off -- it is underspecified to the point of causing harm, and I'm shocked when I hear about people using it in production. I just use Envoy and have an xDS layer to integrate with Kubernetes, and it does exactly what it should do, and no more. (I would like some DNS IaC though.)

Many things associated with Kubernetes are imperfect, like Gitops. A lot of people have trouble with the stack that pushes software to production, and there should be some sort of standard here. (I use ShipIt, a Go program to edit manifests https://github.com/pachyderm/version-bump, and ArgoCD, and am very happy. But it was real engineering work to set that up, and releasing new versions of in-house code is a big problem that there should be a simple solution to.)

Most of these things are not problems brought about by Kubernetes, of course. If you just have a Linux box, you still have to configure auth and observability. But also, your website goes down when the power supply in the computer dies. So I think Kubernetes is an improvement.

The thing that will kill Kubernetes, though, is Helm. I'm out of time to write this comment but I promise a thorough analysis and rant in the future ;)

Re:Kubernetes is our generation's Multics by randomswede on Friday July 23, 2021 @03:36AM (#27928191)
Helm's biggest problem is...

Let me rephrase that. ONE of Helm's biggest problems is that it uses text-based templating, instead of some sort of templating system that understands the thing it's actually trying to template.

This makes some things much MUCH harder than they should need to be.

It makes it really hard to have your configuration bridge things like "you have this much RAM" or "this is the CPU you have" to flags or environment variables that your code can understand.

It also makes it hard to compose configuration.

As much as I don't like BCL, it is depressingly good at being a job configuration language for "run things in the cloud".

Re:Kubernetes is our generation's Multics by jrockway on Friday July 23, 2021 @08:51PM (#27937455)
I think you actually touch on three good points here. One is that "foo: {{ var }}" is not a hygienic template. If var is equal to "bar\nbaz: quux", you've injected hard-to-debug additional keys into the output. The next is that there are common pieces that Kuberenetes defines, and they are all demoted to map[string]interface{}. For example, a lot of charts have "resources" attached to applications, and those are (in Go land), v1.ResourceRequirements. But it could be anything in Helm, it's just a JSON object. So helm itself can't say "you typed 1000M cpu, but probably meant 1000m cpu". And finally, each chart has total latitude to name anything whatever it wants. One chart could say "myapp: { cpu: 42 }" and another configures that as "yourapp: { resources: { requests: { cpu: 42 } } } }". You get to learn Kubernetes all over every time for each app. With zero documentation, usually, except a values.yaml to cut-n-paste from. (My success rate is low. Every Helm app I've installed has required me to read the source code to get it to do what I want. But, other people have better luck, to be fair.)

On top of all that, the value that Helm delivers to people is "you don't have to read the documentation for Deployment to make a Deployment". But then you have to debug that, and you have another layer of complexity bundled on top of your already weak understanding of the core.

Like I get that Kubernetes asks you a lot of questions just to run a container. But they are all good questions, and the answers are important. Just answer the questions and be happy. (Yes, you need to know approximately how much memory your application uses. You needed to know that in the old pet computer era too -- you had to pick some amount to buy at the memory store. Now it's just a field in a YAML file, but the answer is just as critical. A helm chart can set guesses, and if that makes you feel better, maybe that's the value it delivers. But one day, you'll find the guess is wrong, and realize you didn't save any time.)

Re:Kubernetes is our generation's Multics by randomswede on Monday July 26, 2021 @02:06AM (#27956198)
And crucially, once you have given a resource limit, there's no way to (trivially) feed that back into an environment variable or flag to signal that to the app runtime (which, IIRC, is Really Handy for Java-based apps and can seriously improve the performance of Go-based ones).
Re:Kubernetes is our generation's Multics by dekhn on Wednesday July 21, 2021 @06:34PM (#27913027)
Twice today I had to explain to coworkers than "auth is one of the hardest problems in computer science".

For gRPC and HTTP/2: you're doing end to end gRPC (IE, the TCP connection goes from a user's browser all the way to your backend, without being terminated or proxied)?

Re:Kubernetes is our generation's Multics by jrockway on Thursday July 22, 2021 @05:53PM (#27924829)
I don't think I have raw HTTP/2 streams from user to service anywhere. My preference is to have Envoy in the middle doing routing/statistics, and so the TCP session is not preserved from frontend to backend. Each request/response could be handled by a different backend instance. (I don't think Envoy strictly requires this, however; upgrade/websockets work somehow. But maybe only on HTTP/1.1.) This is generally what people want their load balancer to do; a common complaint is that gRPC opens long-lived streams (channels, actually, using their term), and so one client can overload one backend, when the other 100 replicas could happily handle their request/replies. (gRPC's mechanism for state between requests and replies is server stream/client stream/bidirectional stream, which is different than channels. The individual messages in streams can't be split between backends, and so the load balancer won't interfere with that.)

At work we have a service that communicates to clients over gRPC (the CLI app is a gRPC client). We typically deploy that as two ports on the load balancer, one for gRPC and the other for HTTPS. Again, the TCP connection isn't actually preserved while transiting the load balancer, but it's logically a L4 operation -- one client channel is one server channel. If the backend becomes unhealthy, you'll have to open a new channel to the load balancer to get a different backend. (This doesn't really come up for us, because people mostly run a single replica of the service.)

Re:Kubernetes is our generation's Multics by parasubvert on Thursday July 22, 2021 @08:57AM (#27918286)
There are some attempts to gradually find alternatives to Helm while remaining compatible with it. See https://carvel.dev/ for example.

There is a lot of innovation possible in this space.

Re:Kubernetes is our generation's Multics by gbrindisi on Wednesday July 21, 2021 @04:36PM (#27911803)
> The thing that will kill Kubernetes, though, is Helm. I'm out of time to write this comment but I promise a thorough analysis and rant in the future ;)

Too much of a cliffhanger! Now I want to know your pow :)

Re:Kubernetes is our generation's Multics by throwdbaaway on Wednesday July 21, 2021 @08:29PM (#27913923)
Ever since Microsoft acquired the company behind Helm and https://news.ycombinator.com/item?id=11922299 (try clicking the article link), it has been used as a showcase when onboarding azure customers, to somehow prove that "yeah azure is hip and we love open source".

So, yes, we need to know.

Re:Kubernetes is our generation's Multics by akvadrako on Wednesday July 21, 2021 @06:42PM (#27913095)
I don't know why anyone uses Helm. I've done a fair amount of stuff with k8s and never saw the need. The builtin kustomize for is simple and flexible enough.
Re:Kubernetes is our generation's Multics by johnsoft on Wednesday July 21, 2021 @09:05PM (#27914164)
I use Helm because I haven't found another tool that deletes resources in the cluster when I delete them from the yaml. kubectl --prune is unstable and super buggy. I would love to ditch Helm. Is there a tool I should know about that covers this?
Re:Kubernetes is our generation's Multics by parasubvert on Thursday July 22, 2021 @08:58AM (#27918298)
Take a look at Kapp on https://carvel.dev/ for this, possibly.
Re:Kubernetes is our generation's Multics by magnushiie on Wednesday July 21, 2021 @10:57PM (#27914802)
kpt live apply prunes resources
Re:Kubernetes is our generation's Multics by Filligree on Wednesday July 21, 2021 @03:41PM (#27911089)
Ditto.

Granted, I have to assume that borg-sre, etc. etc. are doing a lot of the necessary basic work for us, but as far as the final experience goes?

95% of cases could be better solved by a traditional approach. NixOps maybe.