News for hackers. Stuff that matters.
Re:Paragon submits 27k-line NTFS driver to Linux kernel by nitrogen on Sunday August 23, 2020 @08:00PM (#24256358)
GUI file copy tools should be using O_DIRECT, or periodically calling f/sync(). An argument could also be made that the kernel write cache should have a size limit so that one-off write latency is masked, but very slow bulk I/O is not masked.
Re:Paragon submits 27k-line NTFS driver to Linux kernel by wtallis on Sunday August 23, 2020 @09:10PM (#24256725)
O_DIRECT seems like overkill, and the lack of write buffering could be a real detriment in some circumstances. Syncing at the end of each operation (from the user's perspective) should be the best mix of throughput and safety, but it makes it hard to do an accurate progress bar. Before the whole batch operation is finished, it may be useful to periodically use madvise or posix_fadvise to encourage the OS to flush the right data from the page cache—but I don't know if Linux really makes good use of those hints at the moment.

On really new kernels, it might work well to use io_uring to issue linked chains of read -> write -> fdatasync operations for everything the user wants to copy, and base the GUI's progress bar on the completion of those linked IO units. That will probably ensure the kernel has enough work enqueued to issue optimally large and aligned IOs to the underlying devices. (Also, any file management GUI really needs to be doing async IO to begin with, or at least on a separate thread. So adopting io_uring shouldn't be as big an issue as it would be for many other kinds of applications.)

Re:Paragon submits 27k-line NTFS driver to Linux kernel by im3w1l on Sunday August 23, 2020 @09:34PM (#24256825)
If you syncfs every reasonable-unit-of-time, you can get a progress bar.
Re:Paragon submits 27k-line NTFS driver to Linux kernel by wtallis on Sunday August 23, 2020 @09:47PM (#24256882)
Not always. If you're reading from a SSD and writing to a slow USB 2.0 flash drive, you could end up enqueuing in one second a volume of writes that will take the USB drive tens of seconds to sync(), leading to a very unresponsive progress bar. You almost have to do a TCP-like ramp up of block sizes until you discover where the bottleneck is.