News for hackers. Stuff that matters.
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by kstrauser on Tuesday July 02, 2024 @09:46AM (#40856611)
What’s wrong about it? I haven’t managed k8s much and don’t have enough experience to evaluate their claim.
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by potoftea on Tuesday July 02, 2024 @09:58AM (#40856762)
I can't comment about about other statement.

But there are serious drawbacks if you don't set requests/limits for mission-critical process, is that they can be killed by kernel to free some resources (if the node reach max resource usage)

When you don't set cpu/memory limit your pod QoS class is burstable, which better then BestEffort, but still get assigned `oom_score_adj` score. IMHO you almost 99% you want `Guaranteed` for critical process.

1. oom_score_adj - https://kubernetes.io/docs/concepts/scheduling-eviction/node...

2. Guaranteed - https://kubernetes.io/docs/tasks/configure-pod-container/qua...

3. QoS - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/

Re:For the Love of God, Stop Using CPU Limits on Kubernetes by lantry on Tuesday July 02, 2024 @10:02AM (#40856795)
The post recommends setting both request and limit for memory. It's only for CPU that it says to use a request and not a limit. It's my understanding that the kernel only kills processes when it's out of memory. In the docs you linked, there doesn't seem to be an "eviction signal" for CPU.
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by potoftea on Tuesday July 02, 2024 @10:08AM (#40856855)
This doens't match what I've read.

IF you don't set limit to cpu & memory, your pod QoS will be burstable and from kernel/cgroup perspective this process can be killed - it's only question of oom_score_adj.

OOMScore from linux - https://www.freedesktop.org/software/systemd/man/latest/syst...

Re:For the Love of God, Stop Using CPU Limits on Kubernetes by cassianoleal on Tuesday July 02, 2024 @12:59PM (#40858463)
OOM means "Out Of Memory". The post is about CPU.

> Everything in this post is about CPU and not memory.

Re:For the Love of God, Stop Using CPU Limits on Kubernetes by potoftea on Tuesday July 02, 2024 @04:57PM (#40860334)
I am aware of it, are you aware that Kubernetes assigned Burstable Qos if you don't defined CPU requests and limits?

What I'm trying to say that while try to avoid one pitfall you fall right into an other - in short there are good reasons to use cpu limits.

Re:For the Love of God, Stop Using CPU Limits on Kubernetes by cassianoleal on Tuesday July 02, 2024 @07:14PM (#40861353)
That's not what you're saying though. You're also putting requests/limits for CPU and memory in the same class, which they're not.

Lack of CPU will never cause a pod to be killed. Evicted, yes. Killed, no.

Lack of memory can cause a pod to be OOM killed. That's an entirely different failure mode.

Eviction should always be expected by your pods. They should be able to handle eviction gracefully. On the other hand, there is nothing you can do to handle an OOM kill gracefully.

Re:For the Love of God, Stop Using CPU Limits on Kubernetes by candiddevmike on Tuesday July 02, 2024 @10:37AM (#40857130)
K8s works best (economically) when it can bin pack things. The only way it can bin pack things safely is by having user provided limits--its not smart enough to right size your app (out of the box at least). Not setting them means you're going to end up paying more or have resource contention/outages.

It's better to set the limits higher than you need than to not set them at all. Ideally this is easily done since you're profiling/load testing your app and you understand the appropriate sizing for it, right?

Re:For the Love of God, Stop Using CPU Limits on Kubernetes by sadops on Tuesday July 02, 2024 @11:15AM (#40857460)
Ha! Always end with a joke, I love it.
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by jiehong on Tuesday July 02, 2024 @10:45AM (#40857210)
Unless your app can use all the cpu you assign it
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by cassianoleal on Tuesday July 02, 2024 @01:02PM (#40858492)
The problem is that the Kubernetes CPU scheduler is... not great.

There are many instances where it will throttle CPU of a pod way before it even reaches the requested amount, let alone the limits. This is only true if you set limits. If you don't, it will always have the requested amount available, plus whatever can be spared that's not in use by other processes. It will throttle at most down to the requested amount, and in most cases not at all.

Re:For the Love of God, Stop Using CPU Limits on Kubernetes by crest on Tuesday July 02, 2024 @10:34AM (#40857101)
Users come to expect the performance that was never promised or even properly requested. Once you efficiently load the system they complain because you overdelivered and they got used to it.
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by tardy_one on Tuesday July 02, 2024 @10:56AM (#40857303)
We've been fixing that over time and now no one expects a computer to be as fast as it accidentally was in 2010.
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by sadops on Tuesday July 02, 2024 @11:16AM (#40857468)
Why, by 2030, computers ought to be even slower!
Re:For the Love of God, Stop Using CPU Limits on Kubernetes by sadops on Tuesday July 02, 2024 @11:15AM (#40857464)
I'm sorry, in what way are modern servers not fast enough to serve a few hundred CRUD requests per second?