News for hackers. Stuff that matters.
Re:Comparison of Rust async and Linux thread context switch ... by khuey on Thursday February 11, 2021 @11:21PM (#26110706)
io_uring will help a lot here.
Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Thursday February 11, 2021 @11:28PM (#26110743)
Well, it can, but not always. Remember that if you’re waiting for an event to arrive, that generally involves a syscall, the thread being put to sleep, and then the thread being woken up. Any time you’re doing that, think, “Could I just replace this polling system with a call to read()?”

What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Whether you are often on the fast path for io_uring will depend on the particulars of your application and its I/O patterns.

Re:Comparison of Rust async and Linux thread context switch ... by scottlamb on Friday February 12, 2021 @12:28AM (#26111012)
> What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more.

Isn't "not on the fast path any more" a bit absolutist? io_uring's "slow" path is roughly one syscall per iteration, right? That's still many fewer syscalls than one syscall per IO operation (or more if any return EAGAIN/EWOULDBLOCK) as you'd be doing without it. I'm not sure I really care about eliminating that last syscall per iteration; it seems minor in comparison.

Re:Comparison of Rust async and Linux thread context switch ... by volta83 on Friday February 12, 2021 @03:21AM (#26111669)
> io_uring's "slow" path is roughly one syscall per iteration, right?

Right, so if a blocking API makes 1 syscall, io_uring would make N syscals for N iterations.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @04:24AM (#26111917)
Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough).

That's pretty huge amortization. Rough benchmarks we've done are showing double throughput for io_uring for 4096 byte AF sector write/fsync/read combos: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...

Re:Comparison of Rust async and Linux thread context switch ... by volta83 on Friday February 12, 2021 @07:40AM (#26113004)
> Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough).

Same for the blocking case. If I do a syscall to read a whole file, its just 1 syscall creating millions of I/O operations.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @10:34AM (#26114755)
Sure, but that's all you'll ever do with the blocking case: 1 syscall at a time, while your program sits and does nothing with the CPU, whereas with io_uring at least you can do CPU while you wait on your IO. So even ignoring the IORING_SQPOLL option that requires no io_uring_enter() syscall, a basic usage of io_uring is still going to be faster.

io_uring is a bicycle for IO, and you can ride it as fast as you want to. But it's apples and oranges to blocking IO, which is always stuck in first gear.

Re:Comparison of Rust async and Linux thread context switch ... by volta83 on Friday February 12, 2021 @11:02AM (#26115068)
> while your program sits and does nothing with the CPU

The CPU can run other threads while the hardware does DMA transfers. The thread just yields when the transfer is started, and a hardware exception wakes it up when the DMA transfer finishes.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @12:22PM (#26116032)
Sure, but we're comparing the efficiency of one of your program's single threads, because otherwise you could take that same argument you just used and turn it around and say fine, just run another thread then with another io_uring... and you're still ahead. You have to compare at the smallest unit of control plane.

At the same time, multiple threads for a single program introduce context switches which are becoming horrendously expensive compared to the sheer number of IOPS that modern NVMe SSDs can do.

Thread-per-core designs built around io_uring are the future of IO on Linux.

Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Friday February 12, 2021 @12:34AM (#26111024)
io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall.

I am a bit baffled how this could possibly be considered an “absolutist” viewpoint—I am just saying that there exist scenarios where io_uring is not helpful. This should be uncontroversial.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @04:32AM (#26111944)
> io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall.

That's not correct, io_uring was "absolutely" designed, at least in the technical sense, for zero syscalls in the slow path (if you want to):

  IORING_SETUP_SQPOLL
  When this flag is specified, a kernel thread is created to perform submission queue polling.
  An io_uring instance configured in this way enables an application to issue I/O without ever
  context switching into the kernel. By using the submission queue to fill in new submission
  queue entries and watching for completions on the completion queue, the application can submit
  and reap I/Os without doing a single system call.
From the man page: https://manpages.debian.org/unstable/liburing-dev/io_uring_s...

This mode required privileges in early kernel versions but that's already changed. Things are moving fast.

Re:Comparison of Rust async and Linux thread context switch ... by scottlamb on Friday February 12, 2021 @12:47AM (#26111078)
You mean when there's only one thing to do per iteration? I'd describe that as when mostly idle. As the system gets more loaded, the one syscall per iteration matters less and less.
Re:Comparison of Rust async and Linux thread context switch ... by quotemstr on Thursday February 11, 2021 @11:45PM (#26110818)
io_uring is specifically designed so that zero system calls are necessary while the system is busy. Userspace and the kernel both update ring buffers, and ring buffers can be checked and drained without entering the kernel at all.
Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Friday February 12, 2021 @12:03AM (#26110915)
Yes, that’s what “if you haven’t received new events when you poll” means.
Re:Comparison of Rust async and Linux thread context switch ... by the8472 on Friday February 12, 2021 @05:41AM (#26112246)
io_uring doesn't mean async though. You can also use it for blocking batch execution of syscalls. E.g. when you need to stat hundreds of files or wait for several child processes at once. So with some batch-oriented convenience wrappers it can help threaded code too.