If I understand this correctly one can use RWF_NOWAIT to do what one would naively expect from O_NONBLOCK. Am I correct?
How does this differ from recv(..., MSG_DONTWAIT)?
Besides some technical reasons, the big reason was an "impedance mismatch" of the recv API which was working with stream data. In that context you're depending on another party send data. Also, you can wait on this to happen using another API (select et. al) so the buffer will be filled not by your actions. On the other hand preadv2(..., RWF_NOWAIT) is not going to trigger any more read in and theres no wait to wait on the data. Although the previous statement is not a 100% true... preadv2 may or may not trigger readahead (if it's enabled).
Here's the whole thread about it... if you're interested in the history of how this came to be: https://lkml.org/lkml/2014/7/24/787
I would assume the intent is similar but applied to a different type/source of IO.
This lets you play around with policy for blocking reads in user space. Examples: try to make progress on partially read data & queue up the rest on another thread; try performing reads from your network thread and if unavailable queue up the blocking read onto another disk thread & serve a different request.
There's no RWF_NOWAIT support for write in pwritev2. But it's technically possible to implement it and if your write will cause a writeout to disk return EWOULDBLOCK.
LWN description: https://lwn.net/Articles/612483/
LWN summary of my talk: https://lwn.net/Articles/636967/
Final patch set by Christoph: https://lwn.net/Articles/731700/
Disclaimer: I'm the author of the preadv2/pwritev2 syscalls. And the original author of the support for RWF_NOWAIT, which Christoph Hellwig took over. So feel free to consider this to be self congratulatory.
Correction: It looks like there's a patch set floating around for adding support for RWF_NOWAIT in pwritev2 as well. https://patchwork.kernel.org/patch/9787271/ (comment from Christoph)