It also provides cross-platform aligned buffers for DMA.
Note that block size can be logical or physical sector size. IIRC st_blksize will typically report logical sector size (usually 512 bytes), but physical sector size (usually 4096 bytes) is often larger for Advanced Format drives, and is what you want to be using for optimal alignment. Any multiple of 4096 bytes is usually a safe bet for block size. Note that some SSD drives have 8192 or 16384 bytes as a "physical sector" page size / block size.
In the benchmark what was the physical block size, 16KB? it seems with O_DIRECT after 16KB the speed tapers off and stays at about 130MB/s. And there is a significant jump from 4K (60Mb/s) and 8K (90Mb/s).
Thanks for the input, glad you found it useful!
Regarding the benchmark, the physical block size of the drive itself was 512 bytes. The memory page size on the Ubuntu system was 4K. It would be good to update the benchmark script to include these kinds of numbers.
I'm not sure why O_DIRECT speed increased with block size=4K (60MB/s) to 8K (90MB/s) and tapered off at 16KB (130MB/s). Perhaps system call overhead is more noticeable at 4K and 8K and less so at 16KB? Any other ideas?
130MB/s is the raw drive write speed and matches what I got from dd for the same flags for the same drive.
(vm)splice maybe?
Noticed this bit "block size which is typically 512 or 4096 bytes" and was wondering how would the application know how to align. Does it query the file descriptor for block size? Is there an ioctl call for that?
When it comes to IO it's also possible to differentiate between blocking/non-blocking and synchronous/asynchronous, and those categories are orthogonal in general.
So there is blocking and synchronous: read, write, readv, writev. The calling thread blocks until data is ready and while it gets copied to user memory.
non-blocking synchronous: using non-blocking file descriptors with select/poll/epoll/kqueue. Checking when data is ready is done asynchronously but then read, write still happens inline and the thread waits for it to be copied to user space. This works for socket IO in Linux but not for disk.
non-blocking asynchronous: using AIO on Linux. Here both waiting till data is ready to be transferred and transferring happens asynchronously. aio_read returns right way before read finished. Then have to use aio_error to check for its status. This works for disk but not socket IO on Linux.
blocking asynchronous: nothing here