david_xia
This means
1. we're using a kernel that doesn't contain the patch. 2. the patch wasn't sufficient to prevent unnecessary CPU throttling 3. latency is caused by something other than CPU throttling
To rule out 1, I again checked that our GKE clusters (which are using nodes with Container Optimized OS [COS] VM images) are on a version that contains the CFS patch.
```
dxia@one-of-our-gke-nodes ~ $ uname -a Linux one-of-our-gke-nodes 4.19.112+ #1 SMP Sat Apr 4 06:26:23 PDT 2020 x86_64 Intel(R) Xeon(R) CPU @ 2.30GHz GenuineIntel GNU/Linux
```
Kernel version is 4.19.112+ which is a good sign. I also checked the COS VM image version.
gke-11512-gke3-cos-77-12371-227-0-v200605-pre
The cumulative diff for [COS release notes][2] for cos-stable-77-12371-227-0 show this lineage (see "Changelog (vs ..." in each entry).
cos-stable-77-12371-227-0 77-12371-208-0 77-12371-183-0 77-12371-175-0 77-12371-141-0 <- This one's notes say "Fixed CFS quota throttling issue."
Now looking into 2:
This dashboard [5]. Top graph shows an example Container's CPU limit, request, and usage. The bottom graph shows the number of seconds the Container was CPU throttled as measured by sampling the local kubelet's Prometheus metric for `container_cpu_cfs_throttled_seconds_total` over time. CPU usage data is collected from resource usage metrics for Containers from the [Kubernetes Metrics API][6] which is returns metrics from the [metrics-server][7].
The first graph shows usage is not close to the limit. So there shouldn't be any CPU throttling happening.
The first drop in the top graph was decreasing the CPU limit from 24 to match the CPU requests of 16. The decrease of CPU limit from 24 to 16 actually caused CPU throttling to increase. We removed CPU limits from the Container on 8/31 12:00 which decreased number of seconds of CPU throttling to zero. This makes me think the kernel patch wasn't sufficient to prevent unnecessary CPU throttling.
This K8s Github issue ["CFS quotas can lead to unnecessary throttling #67577"][8] is still open. The linked [kernel bug][9] has a comment saying it should be marked fixed. I'm not sure if there are still CPU throttling issues with CFS not tracked in issue #67577 though.
Because of the strong correlation in the graphs between removing CPU limits and CPU throttling, I'm assuming the kernel patch named "Fixed CFS quota throttling issue." in COS 77-12371-141-0 wasn't enough.
Questions
1. Anyone else using GKE run into this issue?
2. Does anyone have a link to the exact kernel patch that the COS entry "Fixed CFS quota throttling issue." contains? A Linux mailing list ticket or patch would be great so I can see if it's the same patch that various blog posts reference.
3. Anyone aware of any CPU throttling issues in the current COS version and kernel we're using? 77-12371-227-0 and 4.19.112+, respectively.
[1]: https://medium.com/omio-engineering/cpu-limits-and-aggressiv...
[2]: https://cloud.google.com/container-optimized-os/docs/release...
[5]: https://share.getcloudapp.com/o0u8KoEn
[6]: https://kubernetes.io/docs/tasks/debug-application-cluster/r...
[7]: https://github.com/kubernetes/kubernetes/tree/master/cluster...
[8]: https://github.com/kubernetes/kubernetes/issues/67577
Equifax C.E.O. Richard Smith Is Out After Huge Data Breach
Uber internal investigation report by Covington and Burling
What a liberal sociologist learned from spending five years in Trump's America
Ghosts of White People Past: Witnessing White Flight from an Asian Ethnoburb
Tim Cook Opposes Order for Apple to Unlock iPhone, Setting Up Showdown
Euclid the game
Control flow in brainfuck
How's My SSL?
The Theranos controversy, explained
How Disney associates its villains with cross gender appearances
Alabama police officer badly injures Indian grandfather during sidewalk stop
Dizzying but invisible depth
Leaving Chef because of death threats for open source contributions
Investors reportedly buying into Snapchat at $10B valuation
I was thinking very generally of just having the device specify an HTTP(S) proxy and having that remote server be able to inspect any HTTP(S) traffic whether its from a browser or native apps.
Ask HN: Is there a web app to inspect network requests from any device?
Desktop browsers let you inspect network traffic via a developer console. For mobile devices you can install software like Charles proxy onto your local computer and proxy HTTP and HTTPS traffic from your phone through that computer and inspect there.Is there a web app that does what Charles proxy does? If not, why? Would a web proxy that allows developers to see network requests sent by their...
The Cyber Security Mess
TurboTax "Analyzes Your Tax Return"
Vim + tmux + Cloud Virtual Machine = Increased Productivity
PHP namespaces are flawed
Resources for Learning Vim
"Maybe we go out in order to fall short . . . because we want to learn how to be good at being people . . . and moreover, because we want to be people."
She's saying that addiction to people is not the same as addiction to cigarettes. This is a good kind of addiction because human beings are by nature social creatures and should interact with one another.