I'm okayish at systems level debugging but I never figured that one out. It caused the kernel to use alot of memory. Arguable whether or not it impacted performance since it was a cache.
Edit: apparently it has been done, but only very recently and hasn't had a major distro ship it yet
There was a kernel patch though that solved around 95% of issues with no userspace changes: vm.dirty_write_behind. Unfortunately it never made it into the mainline kernel. [1] For strong latency guarantees it ws insufficient but it greatly improved the typical network/IO spike alternations described above.
I'm surprised it was never fixed upstream despite that fact that even most basic and simple scenarios like "nginx writes a log file to the disk" sometimes explode with seconds-long lockups on memory-fat machines.
[1] https://lore.kernel.org/linux-fsdevel/156896493723.4334.1334...
That is unfortunate, and a bit surprising given how positively both Linus Torvalds, and Jens Axboe, responded to the patch.
But really, this is all fantasy given that I know nothing about the virtual memory internal data structures.
Edit: fixed typo
[1] https://docs.kernel.org/core-api/xarray.html
[2] https://elixir.bootlin.com/linux/latest/source/include/linux...
[3] https://elixir.bootlin.com/linux/latest/source/mm/page-write...
[4] https://elixir.bootlin.com/linux/latest/source/mm/page-write...
There is just one caveat with regards to the previous solution: pages that are already pending writeout still need to be locked. In the previous design, a write to a page already pending writeout could be duplicated into the new dirty queue. But since it seems that a page can exist only once in the page cache, it cannot be written to without possibly violating write barriers. So the write must block until the writeout process has un-dirtied the page.
Armchair programming is fun, isn't it? :)
A better design would be to only lock a subset of dirty pages and let new writes to virtual memory continue while the write-out is happening, but it doesn't seem like the system can accomodate that.
A simple test to see this in action is to use netcat to transfer a large file across the network, and monitor the device activity with sar or atop (or just check the disk/network activity lights). What you'll see is that while the disk is writing, network activity drops to zero, and when network activity resumes, the disk remains idle again for seconds. It doesn't matter how much smaller you make vm.dirty_background_{bytes,ratio} with respect to vm.dirty_{bytes,ratio}, the network traffic will block as soon as the "background" disk write starts. The only effect a low value for vm.dirty_background_{bytes,ratio} has is to increase the frequency at which the network and disk activity will alternate.