News for hackers. Stuff that matters.
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by tremon on Wednesday December 13, 2023 @01:52PM (#38632306)
Same goes for I/O scheduling. The virtual memory subsystem seems to only have two modes: buffering and write-out. The system will buffer writes until some threshold is reached, then it switches to write-out mode and flushes all dirty pages regardless of their age. The problem is that new pages are locked during write-out, so pending writes will stall, regardless of the memory pressure on the rest of the system.

A better design would be to only lock a subset of dirty pages and let new writes to virtual memory continue while the write-out is happening, but it doesn't seem like the system can accomodate that.

A simple test to see this in action is to use netcat to transfer a large file across the network, and monitor the device activity with sar or atop (or just check the disk/network activity lights). What you'll see is that while the disk is writing, network activity drops to zero, and when network activity resumes, the disk remains idle again for seconds. It doesn't matter how much smaller you make vm.dirty_background_{bytes,ratio} with respect to vm.dirty_{bytes,ratio}, the network traffic will block as soon as the "background" disk write starts. The only effect a low value for vm.dirty_background_{bytes,ratio} has is to increase the frequency at which the network and disk activity will alternate.

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by alyandon on Wednesday December 13, 2023 @02:11PM (#38632605)
Ran into similar pathological behavior on servers with huge amounts of ram (>1 TiB). Negative dentries builds up until the kernel decides it is time to reclaim and locks up the server for >60 seconds scanning pages. :-/
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by anonacct37 on Wednesday December 13, 2023 @04:51PM (#38634664)
I also ran into spooky negative dentry issues. On a fleet of seemingly identical physical machines one machine would constantly check for files that didn't exist (iirc it was somehow part of nss and a quick Google search seems to confirm https://lwn.net/Articles/894098/ ).

I'm okayish at systems level debugging but I never figured that one out. It caused the kernel to use alot of memory. Arguable whether or not it impacted performance since it was a cache.

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by tremon on Wednesday December 13, 2023 @02:20PM (#38632761)
Yes, indeed. The more ram available, the more pages are available as buffer cache, the more pronounced the system stalls are. Though in my example, the determining factor is available_memory/disk_write_troughput, while in your example it is available_memory/single_core_throughput (assuming the page scan is a single-thread process).
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by mkhnews on Wednesday December 13, 2023 @07:12PM (#38636109)
Does disabling transparent huge pages help any here ?
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by tremon on Thursday December 14, 2023 @01:38PM (#38645207)
Have not tried it, but I would not expect any difference. FAFAIK, transparent huge pages are only about organizing free/unused/allocated memory; in both mine and the parent's example, the issue isn't about memory management per se but about synchronous operations on data taking too long (I/O write in my case, cache cleanup for the GP).
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by freedomben on Wednesday December 13, 2023 @02:09PM (#38632572)
Why do you think it's still this way? Why hasn't somebody implemented something better? Is it too core that we can't afford any instability?

Edit: apparently it has been done, but only very recently and hasn't had a major distro ship it yet

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by ahartmetz on Wednesday December 13, 2023 @02:22PM (#38632798)
Oh? That sounds important and interesting. Do you have a URL or commit hash with more information?
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by freedomben on Wednesday December 13, 2023 @02:28PM (#38632911)
I'm very confused now so I have no idea what to think. I need to take some time to read through everything, but in the mean time here are some relevant links I've seen:

https://lwn.net/Articles/925371/

https://lkml.org/lkml/2016/4/23/135

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by ahartmetz on Wednesday December 13, 2023 @03:01PM (#38633411)
Hm yes, maybe you are confused. The articles you posted seem to be about CPU, not I/O.
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by alexey-salmin on Wednesday December 13, 2023 @05:08PM (#38634880)
A complete nightmare. In the past I had to work on an IO-heavy system with 1-5ms read-latency guarantees and the only working solution was to completely manage all writes in the userspace. Manual buffering, manual throttling, manual fallocate, then either DIRECT_IO or mmap+sync_file_range depending on whether you need the written data to be readily available for reads or not.

There was a kernel patch though that solved around 95% of issues with no userspace changes: vm.dirty_write_behind. Unfortunately it never made it into the mainline kernel. [1] For strong latency guarantees it ws insufficient but it greatly improved the typical network/IO spike alternations described above.

I'm surprised it was never fixed upstream despite that fact that even most basic and simple scenarios like "nginx writes a log file to the disk" sometimes explode with seconds-long lockups on memory-fat machines.

[1] https://lore.kernel.org/linux-fsdevel/156896493723.4334.1334...

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by tux1968 on Wednesday December 13, 2023 @07:50PM (#38636388)
> Unfortunately it never made it into the mainline kernel.

That is unfortunate, and a bit surprising given how positively both Linus Torvalds, and Jens Axboe, responded to the patch.

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by viraptor on Thursday December 14, 2023 @12:26AM (#38638115)
Would a solution to this include some kind of new queue? It is the fsync barrier enough for synchronising the writes? I'm thinking of situations where you try to write a subset of blocks, but you end up writing some new ones before old ones.
Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by tremon on Thursday December 14, 2023 @12:50PM (#38644474)
I'm not sure how the current vm system tracks dirty pages, so I wouldn't know where to start suggesting realistic solutions. But yes, assuming the dirty pages are organized in some sort of specialized data structure (queue or otherwise), you could do this without needing new barriers: it would suffice to create a new dirty-page root structure before starting writeout, and redirect freshly dirtied pages to the new root. The writer process can then take its leisurely time committing the pending writes to disk without obstructing other processes issuing new writes. New writes to pending pages can be reparented to the new root via an RCU mechanism (to not violate previous write barriers), and incomplete writes can be reparented by the writer process in a similar way. When the writeout process completes, the old root can be completely discarded; the writer could even consume and free the old structure while traversing it.

But really, this is all fantasy given that I know nothing about the virtual memory internal data structures.

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by afr0ck on Friday December 15, 2023 @10:24AM (#38655033)
The kernel page cache is an XArray [1] (data structure here [2]) and dirty page pointers are annotated with the dirty tag when the page is dirtied [3]. There is a per-filesystem callback that usually always ends up here [4], where you can see the the above data structure [2] entry for that page is being tagged as dirty.

Edit: fixed typo

[1] https://docs.kernel.org/core-api/xarray.html

[2] https://elixir.bootlin.com/linux/latest/source/include/linux...

[3] https://elixir.bootlin.com/linux/latest/source/mm/page-write...

[4] https://elixir.bootlin.com/linux/latest/source/mm/page-write...

Re:The Linux Scheduler: A Decade of Wasted Cores (2016) [pdf] by tremon on Friday December 15, 2023 @03:50PM (#38658631)
Thanks. So there is no "specialized data structure" for tracking dirty pages, the page dirty status is tracked within the regular page cache hierarchy. And the obvious in-line solution (use a wraparound counter for tracking dirty state, increase dirty values for new pages while writing out lower values) isn't easy because the tag is a bitfield. So the easiest solution would be to have two separate dirty markers (e.g. dirty_a and dirty_b), and have the writeout process toggle the new-dirty marker from A to B so it can safely write dirty_a pages without interference. And vice-versa for the next writeout.

There is just one caveat with regards to the previous solution: pages that are already pending writeout still need to be locked. In the previous design, a write to a page already pending writeout could be duplicated into the new dirty queue. But since it seems that a page can exist only once in the page cache, it cannot be written to without possibly violating write barriers. So the write must block until the writeout process has un-dirtied the page.

Armchair programming is fun, isn't it? :)