Re:Paragon submits 27k-line NTFS driver to Linux kernel
by nitrogen on Sunday August 23, 2020 @08:00PM
(#24256358)
GUI file copy tools should be using O_DIRECT, or periodically calling f/sync(). An argument could also be made that the kernel write cache should have a size limit so that one-off write latency is masked, but very slow bulk I/O is not masked.
Re:Paragon submits 27k-line NTFS driver to Linux kernel
by wtallis on Sunday August 23, 2020 @09:10PM
(#24256725)
O_DIRECT seems like overkill, and the lack of write buffering could be a real detriment in some circumstances. Syncing at the end of each operation (from the user's perspective) should be the best mix of throughput and safety, but it makes it hard to do an accurate progress bar. Before the whole batch operation is finished, it may be useful to periodically use madvise or posix_fadvise to encourage the OS to flush the right data from the page cache—but I don't know if Linux really makes good use of those hints at the moment.
On really new kernels, it might work well to use io_uring to issue linked chains of read -> write -> fdatasync operations for everything the user wants to copy, and base the GUI's progress bar on the completion of those linked IO units. That will probably ensure the kernel has enough work enqueued to issue optimally large and aligned IOs to the underlying devices. (Also, any file management GUI really needs to be doing async IO to begin with, or at least on a separate thread. So adopting io_uring shouldn't be as big an issue as it would be for many other kinds of applications.)
Re:Paragon submits 27k-line NTFS driver to Linux kernel
by im3w1l on Sunday August 23, 2020 @09:34PM
(#24256825)
If you syncfs every reasonable-unit-of-time, you can get a progress bar.
Re:Paragon submits 27k-line NTFS driver to Linux kernel
by wtallis on Sunday August 23, 2020 @09:47PM
(#24256882)
Not always. If you're reading from a SSD and writing to a slow USB 2.0 flash drive, you could end up enqueuing in one second a volume of writes that will take the USB drive tens of seconds to sync(), leading to a very unresponsive progress bar. You almost have to do a TCP-like ramp up of block sizes until you discover where the bottleneck is.
Re:Paragon submits 27k-line NTFS driver to Linux kernel
by flukus on Sunday August 23, 2020 @11:14PM
(#24257256)
Which distro/desktop? My standard ubuntu 18.04 with gnome and mounted through the file manager doesn't do this and copying to a slow USB drive is as glacial as it should be, but copying between internal drives is instant and hidden.
Re:Paragon submits 27k-line NTFS driver to Linux kernel
by wheybags on Monday August 24, 2020 @05:48AM
(#24259011)
Default gnome on Ubuntu 20.04. How much free ram do you normally have? If you don't have enough to buffer the whole operation, then it's not a problem.
Re:Paragon submits 27k-line NTFS driver to Linux kernel
by flukus on Tuesday August 25, 2020 @02:31AM
(#24268628)
Now that I think about it, this might actually explain some bugs I've seen when copying multiple files. Copying one file seems to work but then copying a second the progress sits at 0%, it's probably waiting for the first transfer to sync.
What I end up doing is using a "watch" on some command I can't remember that shows overall dirty pages.