Re:Linux kernel cgroups writeback high CPU troubleshooting
by sirjee on Friday February 14, 2025 @11:59AM
(#43050427)
BTW company-1 == Google and company-2 == FB/Meta.
In addition, Google has completely removed local storage from their servers, so there is no disk I/O at all.
Re:Linux kernel cgroups writeback high CPU troubleshooting
by betaby on Friday February 14, 2025 @12:10PM
(#43050529)
> In addition, Google has completely removed local storage from their servers, so there is no disk I/O at all.
What does that mean? There should be disk somewhere anyway to store gmail messages.
Re:Linux kernel cgroups writeback high CPU troubleshooting
by jeffbee on Friday February 14, 2025 @12:28PM
(#43050699)
https://static.googleusercontent.com/media/sre.google/en//st...
It sounds a bit vanilla on paper, since things like NFS and iSCSI have existed forever.
Re:Linux kernel cgroups writeback high CPU troubleshooting
by sirjee on Friday February 14, 2025 @12:30PM
(#43050722)
Usually Google applications uses high level network storage services like collosus, BigTable or spanner and these high level services are backed by dedicated storage appliances where they bypass the kernel for SSDs and use direct block IO for slow disks. For network, they are moving towards userspace network [1].
[1] https://research.google/pubs/snap-a-microkernel-approach-to-...
I would say it was changed in cgroups v2.
Cgroups v1 was written by a company where only one process on a machine is allowed to do block I/O, and that program is carefully written to not use kernel caches.
Cgroups v2 was written by a company that uses lots of off-the-shelf Linux applications that do ordinary block I/O in the usual naïve way. That's why v2 focuses so much on "pressure".