News for hackers. Stuff that matters.
Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by pm90 on Wednesday November 27, 2024 @01:55AM (#42253608)
There are genuine reasons for running multiple clusters. It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation so they get their own cluster (although somehow its ok that the nodes are still VMs that are probably running on shared nodes… these requirements can be arbitrary sometimes).
Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by yjftsjthsd-h on Wednesday November 27, 2024 @02:09AM (#42253698)
> It helps to sometimes keep stateful (databases generally) workloads on one cluster, have another for stateless workloads etc. Sometimes customers demand complete isolation

Are these not covered by taint/toleration? I guess maybe isolation depending on what exactly they're demanding but even then I'd think it could work.

Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by pm90 on Tuesday December 03, 2024 @11:11PM (#42314427)
Yes only at the node (data plane) level but not the api (control plane).
Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by raverbashing on Wednesday November 27, 2024 @03:29AM (#42254122)
This doesn't make sense to me, and I feel like this is "holding it wrong"

But this is also a caveat with "managing it yourself", you get a lot of people having ideas and shooting themselves in the foot with it

Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by oofbey on Wednesday November 27, 2024 @02:02AM (#42253659)
Honest question - why would you want stateful workloads in a separate cluster from stateless? Why not just use a namespace?
Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by pm90 on Tuesday December 03, 2024 @11:11PM (#42314424)
because namespaces aren’t a failure boundary.

If your api gets hosed, you can create a new cluster and tear down the old one and call it a day.

With a stateful cluster, you can’t do that. As such you put in a lot more care with eg k8s upgrades, or introducing new controllers or admissions/mutating webhooks.

Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by rvense on Wednesday November 27, 2024 @03:05AM (#42254011)
> It helps

Can you elaborate on that? What does it do?

Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by wasmitnetzen on Wednesday November 27, 2024 @05:27AM (#42254822)
Yes, a few, maybe even 10, 12, but 47? It's also a prime number, so it's not something like running each thing three times for dev, stage and prod.
Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by pm90 on Tuesday December 03, 2024 @11:11PM (#42314429)
yeahhhh 47 seems insane
Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by lenkite on Wednesday November 27, 2024 @10:36AM (#42256843)
Most important reason is the ridiculous etcd limit of 8Gb. That alone is the reason for most k8s cluster splits.
Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by mdaniel on Wednesday November 27, 2024 @11:37AM (#42257366)
I hate etcd probably more than most, but that 8Gb seems to just be a warning, unless you have information otherwise https://etcd.io/docs/v3.5/dev-guide/limit/#storage-size-limi...

I'll take this opportunity to once again bitch and moan that Kubernetes just fucking refuses to allow the KV store to be pluggable, unlike damn near everything else in their world, because they think that's funny or something

Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by lenkite on Wednesday November 27, 2024 @05:52PM (#42260611)
It isn't a mere warning. It is strongly recommended as the upper limit.

https://www.perfectscale.io/blog/etcd-8gb

https://github.com/etcd-io/etcd/issues/9771

And yes, I agree not allowing a pluggable replacement is really stupid.

Re:I Stopped Using Kubernetes. Our DevOps Team Is Happier Th... by mdaniel on Wednesday November 27, 2024 @08:23PM (#42261491)
> https://github.com/etcd-io/etcd/issues/9771

> stale bot marked this as completed (by fucking closing it)

Ah, yes, what would a Kubernetes-adjacent project be without a fucking stale bot to close issues willy nilly