News for hackers. Stuff that matters.
Re:Mkfile(8) is severely syscall limited on OS X by dom0 on Sunday February 12, 2017 @07:31AM (#13628059)
And that's why sane people will tell you to not use something like st_blksize as the IO size, because that's far to small. Many tools use 32 KiB, but that can be limiting even on Linux and needlessly uses more CPU (especially if FUSE is involved -- reads are not coalesced at the VFS layer! Read and write merging is done at the block device layer by the IO scheduler there!). Something like 512k-1M is a sane default these days.
Re:Mkfile(8) is severely syscall limited on OS X by gens on Sunday February 12, 2017 @10:14AM (#13628673)
Maybe i'm misunderstanding you, but.. Before every one a "echo 3 > /proc/sys/vm/drop_caches" is executed. Processor is in "performance" mode.

dd if=./big_file.file of=/dev/null bs=64K

535020540 bytes (535 MB, 510 MiB) copied, 8.70128 s, 61.5 MB/s

dd if=./big_file.file of=/dev/null bs=4K

535020540 bytes (535 MB, 510 MiB) copied, 8.36283 s, 64.0 MB/s

dd if=./big_file.file of=/dev/null bs=1K

535020540 bytes (535 MB, 510 MiB) copied, 8.82508 s, 60.6 MB/s

Only thing gone up is the cpu usage (same is negligible at ~60MB/s).

dd if=/dev/zero of=/dev/null bs=1K count=1024K

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 1.06113 s, 1.0 GB/s

dd if=/dev/zero of=/dev/null bs=4K count=256K

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.343595 s, 3.1 GB/s

dd if=/dev/zero of=/dev/null bs=32K count=32K

1073741824 bytes (1.1 GB, 1.0 GiB) copied, 0.160191 s, 6.7 GB/s

Re:Mkfile(8) is severely syscall limited on OS X by dom0 on Sunday February 12, 2017 @01:14PM (#13629629)
At just 60 MB/s it doesn't matter much (even 512 byte blocks would just be a little more than 100k syscalls / second). At a couple hundred MB/s it begins to matter, and with faster NVMe SSDs it's a legit issue.

This becomes a bigger issue if it's not a kernel-based, local FS, but something that does more work, I mentioned FUSE above. Since no read merging happens, small reads like 4k or 32k start to hurt earlier, especially if the FS is written in an interpreter. Some of these also mishandled small reads, incurring an additional penalty (I know because I fixed that in an implementation).

Re:Mkfile(8) is severely syscall limited on OS X by gens on Sunday February 12, 2017 @01:30PM (#13629715)
No point in speculating. I showed you how. Go test it.

Even the fastest SSD is not even near the speed of /dev/zero. Easy to extrapolate any speed from those two points (minus the filesystem overhead (the thing we are talking about is the block layer)).

I'd be interested in a test of a FUSE based fs like ntfs. So if you have one, just run some tests. (don't forget to flush the cache)

Re:Mkfile(8) is severely syscall limited on OS X by brobinson on Sunday February 12, 2017 @12:06PM (#13629240)
Don't forget "oflag=direct"
Re:Mkfile(8) is severely syscall limited on OS X by dom0 on Sunday February 12, 2017 @01:10PM (#13629602)
Uhm, does dd contain all the logic to work with that or is that literally just O_DIRECT |ed into the flags?
Re:Mkfile(8) is severely syscall limited on OS X by cbsmith on Sunday February 12, 2017 @05:54PM (#13631593)
The whole thing is kind of odd. Normally I'd want to leave filling the file to filesystem...