News for hackers. Stuff that matters.
Re:Efficient IO with io_uring [pdf] by zbjornson on Monday May 06, 2019 @06:37PM (#19844648)
I made a prototype of a replacement for Node.js' fs.read/write using io_uring in hopes of it ultimately replacing libuv's current threadpool implementation. Definitely promising! Many props to Jens for his hard work on this and for his helpful discussions on the API.

Benchmarks at [0] and prototype repo at [1]

[0] https://github.com/libuv/libuv/issues/1947#issuecomment-4852... [1] https://github.com/zbjornson/node-io-uring

Re:Efficient IO with io_uring [pdf] by coder543 on Monday May 06, 2019 @06:44PM (#19844699)
io_uring sounds promising.

It'll be interesting to see how long it takes for Tokio to add support for it in the Rust community, and how long it takes for Go to transparently support this instead of a pool of file I/O workers.

Re:Efficient IO with io_uring [pdf] by Matthias247 on Tuesday May 07, 2019 @02:47AM (#19847150)
It will be interesting to see whether Rusts async ecosystem gets support for it. However there is a little bit of a challenge in the form of an impedance mismatch: Rust Futures and thereby also async functions require that an operation is cancellable at any point of time (by dropping the respective Future). IO completion based systems (like io_uring and IOCP) typically don't support this. If the operation in the Rust userspace would be cancelled while the Kernel proceeds with the operation the Kernel would act on invalid memory. There are some workarounds for this, like wrapping the APIs in ones that only used owned buffers that are passed for the whole duration of the operation and reference-counting the IO objects (file descriptors). But the behavior is pretty tricky. [1] describes the problem.

[1] https://github.com/rust-lang-nursery/futures-rs/issues/1278

Re:Efficient IO with io_uring [pdf] by GolDDranks on Tuesday May 07, 2019 @12:32PM (#19851011)
I wonder if it would be possible to pass the ownership of a structure that contains the buffer to kernel, passing a pointer in the user_data. The application could then `std::mem::forget` the buffer, and re-construct it from the raw pointer when the completion event arrives. The Future that originated the IO job could then be dropped independently from the buffer.
Re:Efficient IO with io_uring [pdf] by coder543 on Tuesday May 07, 2019 @11:06AM (#19850168)
That's really interesting. I think some kind of reference counting and a shadow event loop is going to be necessary, if that's the case, so that I/O futures can always be driven to completion even if the developer-facing one is dropped.
Re:Efficient IO with io_uring [pdf] by Matthias247 on Tuesday May 07, 2019 @12:10PM (#19850810)
Yes, reference-counting of outstanding operations and the used buffers is a suitable workaround.

It comes with some downsides though too: The IO operations then only accept certain specialized buffer types and nothing generic like pure byte slices anymore, which might make those things less interoperable. The other issue is that while using refcounting leaves at least the memory in a non-corrupted state the IO source will be in a somewhat undefined state after the cancellation. E.g. if a write operation had been issued on the socket, the API user can not make any assumptions whether or not any byte had been written, and which impact a subsequent write might have. The only reasonable thing to do is to close the IO source (where the close might also need to wait until the still pending background task has completed).

Re:Efficient IO with io_uring [pdf] by coder543 on Tuesday May 07, 2019 @12:13PM (#19850843)
If the future has been dropped, I agree that the source/sink should be closed as soon as safely possible. Usually the resource has to be moved into a future for it to be used, and thus it's logical that no one could continue using it after the future is dropped.
Re:Efficient IO with io_uring [pdf] by the8472 on Tuesday May 07, 2019 @03:43PM (#19852777)
io_uring definitely wants to own the buffers, either for its entire lifetime (with preregistered memory mappings) or at least until completion.

if possible the future should take ownership of the buffer and have it returned on completion.

Re:Efficient IO with io_uring [pdf] by jcranmer on Monday May 06, 2019 @07:44PM (#19845061)
The bug is here, if you want to follow it: https://github.com/tokio-rs/mio/issues/923
Re:Efficient IO with io_uring [pdf] by stbuehler on Tuesday May 07, 2019 @01:15PM (#19851402)
https://github.com/stbuehler/rust-io-uring

Note that io_uring in linux 5.1 has various bugs; especially READ* and WRITE* on sockets/pipe don't work very well (afaik they either block submission or return -EAGAIN with O_NONBLOCK), so the example code in my repo isn't that useful for linux 5.1 (POLL_ADD works fine though).

For proper support in tokio this would have to be done in mio afaict (or tokio would have to replace mio with something else).

Re:Efficient IO with io_uring [pdf] by coder543 on Tuesday May 07, 2019 @01:17PM (#19851426)
do those issues only apply to networking? the big benefit I see for io_uring is that it provides a truly async interface for file I/O for the first time ever in Linux... as far as I can tell.
Re:Efficient IO with io_uring [pdf] by stbuehler on Tuesday May 07, 2019 @01:33PM (#19851593)
I'm not sure about the details for file I/O; files with FMODE_NOWAIT should be fine (block devices, ext4, btrfs, ...).

The idea is that during submission an operation is started with IOCB_NOWAIT; an operation shouldn't block with that. If it returns EAGAIN it will be retried in another kernel thread without IOCB_NOWAIT. Afaik IOCB_NOWAIT support should be announced through FMODE_NOWAIT.

Right now operations that don't support IOCB_NOWAIT ("magic check" in io_file_supports_async [1]) are still run during submission but without the IOCB_NOWAIT flag.

Sockets/pipes don't handle IOCB_NOWAIT, they only care about O_NONBLOCK right now (which is a bug imho).

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

Re:Efficient IO with io_uring [pdf] by emilsedgh on Monday May 06, 2019 @09:27PM (#19845609)
That's a huge improvement! Really looking forward to Node taking advantage of this.
Re:Efficient IO with io_uring [pdf] by mmastrac on Monday May 06, 2019 @10:09PM (#19845786)
If I'm reading your chart correctly, it looks like the thread pool of size 1 is still "beating" the uring, right? What would explain this?
Re:Efficient IO with io_uring [pdf] by BeeOnRope on Monday May 06, 2019 @10:14PM (#19845805)
Lower numbers are better in that chart, so the threadpool size one is only winning on the rightmost part of the plat: just barely at parallelism 16 and slightly more at 32.
Re:Efficient IO with io_uring [pdf] by mmastrac on Monday May 06, 2019 @11:43PM (#19846332)
Makes sense - thanks. Looks like this is a huge win for low parallelism system and a very minor loss at high parallelism.