News for hackers. Stuff that matters.
Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Thursday February 11, 2021 @11:19PM (#26110699)
I am not surprised that the cost of context switching due to I/O readiness can often be roughly equal between async tasks and kernel threads. Normal blocking I/O can be surprisingly efficient because of various factors, such as a reduced need for system calls.

Think about it this way—if you have a user-space thread which wakes up due to I/O readiness, then this means that the relevant kernel thread woke up from epoll_wait() or something similar. With blocking I/O, you call read(), and the kernel wakes up your thread when the read() completes. With non-blocking I/O, you call read(), get EAGAIN, call epoll_wait(), the kernel wakes up your thread when data is ready, and then you call read() a second time.

In both scenarios, you’re calling a blocking system call and waking up the thread later.

Of course, there are scenarios when epoll_wait() returns multiple events, which reduces the number of context switches. But the general result is that it’s not always easy to beat blocking I/O and kernel threads.

Re:Comparison of Rust async and Linux thread context switch ... by singron on Friday February 12, 2021 @12:20AM (#26110981)
Google found that the main cost of context switching isn't really in the syscall boundary but in the task scheduling. That's why their linux fork has optional userspace scheduling of kernel threads with the switchto syscalls[0]. Essentially, if your thread already knows which thread should run next, it can context switch to it without having to schedule in the kernel, which is exactly the situation in these bucket brigade benchmarks.

This benchmark as written is probably underestimating kernel task scheduling cost since only 1 task is runnable at any 1 time, while a realistic multi-threaded system will have more runnable threads to juggle.

0: http://pdxplumbers.osuosl.org/2013/ocw//system/presentations...

Re:Comparison of Rust async and Linux thread context switch ... by otabdeveloper4 on Friday February 12, 2021 @03:35AM (#26111738)
Yes, the point of "async" isn't to save CPU cycles, it's to customize the scheduler so that you can prioritize resource use properly.

(E.g., don't switch to the systemd or sshd thread if a customer's web request is timing out.)

That said, doing this right is out of reach of the average programmer, and it's doubtful that the compiler has enough domain-specific knowledge to do this automatically.

"Async" of the Python and node.js fame is yet another thing, a hack to get around their interpreters' inability to use kernel multitasking features because of global locks.

Re:Comparison of Rust async and Linux thread context switch ... by dmw_ng on Thursday February 11, 2021 @11:36PM (#26110777)
Linux is likely many years from having anything approaching a fully asynchronous system call interface, if anyone was willing to work on it (io_uring makes a huge dent but I don't think it's intending to reimplement everything). Even where async kernel interfaces exist, without reworking of the kernel-internal implementation still there is often the need for a thread for the kernel side to execute on. For example IIRC this is true for swathes of the vfs implementation at present.

So the whole thing is a bit of a false equivalence. Better interfaces that reduce context switches are desirable, but even where they exist often you are just substituting a user thread for a kernel one, and in the general case, there is likely to always be system interfaces that never make it into the brave new world -- take SysV IPC for example (a 1975 era API), it seems doubtful anyone would put the effort into making it async, but there will probably still be times where you might want to consume those interfaces for compatibility or some other obscure reason.

Also consider the case where a user program has a need for some substantial thread pools of its own, it might be the case in some scenarios that reusing resources that must already exist in user space and live in warmed caches makes more sense. Neither async or Linux threads are "better", it will always depend on a particular use case, and even then the right answer might well be some combination of both.

Re:Comparison of Rust async and Linux thread context switch ... by Tuna-Fish on Friday February 12, 2021 @01:50AM (#26111287)
> io_uring makes a huge dent but I don't think it's intending to reimplement everything

At this point it's proponents are being pretty unapologic that it will, in fact, reimplement every part of the syscall interface that is actively used.

Re:Comparison of Rust async and Linux thread context switch ... by lathiat on Thursday February 11, 2021 @11:56PM (#26110876)
I'm not sure about io_uring not reimplementing everything.. it seems to be gaining more and more scope.
Re:Comparison of Rust async and Linux thread context switch ... by throwaway81523 on Friday February 12, 2021 @03:39AM (#26111748)
Another possibility at least in the case of lots of network sockets is DPDK, avoiding almost all the context switches if the user side is async.
Re:Comparison of Rust async and Linux thread context switch ... by continuations on Thursday February 11, 2021 @11:36PM (#26110779)
Then why is it that IO-heavy benchmarks such as the Techempower web benchmark are dominated by async frameworks? The fastest results there are all from async frameworks [1].

And among Rust frameworks the same pattern holds. The fastest Rust frameworks are async while a synchronous frmework such as Rocket is about 20x slower.

[1] https://www.techempower.com/benchmarks/#section=data-r20&hw=...

[2] https://www.techempower.com/benchmarks/#section=data-r20&hw=...

Re:Comparison of Rust async and Linux thread context switch ... by lmm on Friday February 12, 2021 @12:20AM (#26110980)
Those benchmarks measure one very specific scenario: serving lots of small requests concurrently. Async handles that well because that's exactly the scenario where a single epoll_wait() call will return lots of events.
Re:Comparison of Rust async and Linux thread context switch ... by infogulch on Friday February 12, 2021 @02:15AM (#26111373)
Is there a different benchmark that demonstrates a scenario where synchronous syscalls are better suited?
Re:Comparison of Rust async and Linux thread context switch ... by emn13 on Friday February 12, 2021 @03:01AM (#26111576)
Presumably the difference would be smaller or for some frameworks even negative if each request did some actual and not entirely predictable amount of CPU work (e.g. executing some html templating scenario with varying levels of output and perhaps compression), and just in general much more work and using more memory (so the memory overhead is proportionally less relevant), and if the benchmark implementations were not permitted to tune exactly for the workload and system (i.e. so that generalized scheduler defaults are used on both kernel and userspace side). I.e., in a more real-world scenario with all the normal complexities and inefficiencies and development time constraints that are usual.

But yeah, it's be super interesting to actually see that demonstrated - that'd be quite a lot of work, however.

Re:Comparison of Rust async and Linux thread context switch ... by lmm on Friday February 12, 2021 @02:52AM (#26111537)
I doubt you'll find anything as comprehensive and well-presented as the TechEmpower benchmarks, because their particular scenario is one that a lot of frameworks care about competing on (partly because it's difficult enough to be interesting). But I'd expect any benchmark for batch-style processing of large volumes of data would show that.
Re:Comparison of Rust async and Linux thread context switch ... by shihab on Friday February 12, 2021 @06:28AM (#26112510)
Re:Comparison of Rust async and Linux thread context switch ... by volta83 on Friday February 12, 2021 @03:19AM (#26111666)
If your request are huge. For example, imagine you need to read many huge files into memory.

Whether you read one file after the other sequentially, or try to read all of them concurrently, won't make a difference, because your Disk/RAM bandwidth is going to be bottlenecked anyways.

Trying to do this concurrently requires more work that won't pay off, so it might actually be slower.

Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Friday February 12, 2021 @12:13AM (#26110952)
Benchmarks are not everything, and the difference between asynch/synchronous operation is not the only thing the benchmark is testing (each of these different frameworks appear to have their own system for parsing and representing HTTP requests). You should know what usage patterns YOUR application sees, understand the relative cost of engineering time and CPU time for YOUR application, and do tests in YOUR environment.
Re:Comparison of Rust async and Linux thread context switch ... by jacoblambda on Thursday February 11, 2021 @11:43PM (#26110810)
I'd argue that it's because even though blocking IO is cheaper, it's very difficult to maximise performance in a multithreaded/concurrent context.

You could make faster code with it but I wouldn't want to maintain it and you'd have to throw an obscene amount of man hours at it to get that performance.

Re:Comparison of Rust async and Linux thread context switch ... by otabdeveloper4 on Friday February 12, 2021 @03:37AM (#26111744)
> Then why is it that IO-heavy benchmarks such as the Techempower web benchmark are dominated by async frameworks?

Probably because they forgot to enable realtime priority for threads in the synchronous frameworks.

Failing to do that means Linux will starve your web request handling threads in favor of various system tasks you don't care about.

Re:Comparison of Rust async and Linux thread context switch ... by inimino on Friday February 12, 2021 @06:30AM (#26112524)
Rocket is slow because of its design, not because it is insufficiently asynchronous.
Re:Comparison of Rust async and Linux thread context switch ... by Freaky on Friday February 12, 2021 @09:54AM (#26114242)
It's beaten by half a dozen sync Ruby implementations, which should be a pretty good hint that something else is going on.

Lack of HTTP keep-alive is probably the most obvious thing holding it back.

Re:Comparison of Rust async and Linux thread context switch ... by titzer on Friday February 12, 2021 @12:14AM (#26110959)
I mostly agree with you (not the least of which is that blocking I/O is a damn fine API), but the reason that people use async I/O is to have lots of outstanding requests. Typically you would use select (or similar) to service whichever one responds first. That way you can multiplex many I/O streams onto a small number of threads. If threads are memory-intensive, you almost certainly have to do this.
Re:Comparison of Rust async and Linux thread context switch ... by khuey on Thursday February 11, 2021 @11:21PM (#26110706)
io_uring will help a lot here.
Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Thursday February 11, 2021 @11:28PM (#26110743)
Well, it can, but not always. Remember that if you’re waiting for an event to arrive, that generally involves a syscall, the thread being put to sleep, and then the thread being woken up. Any time you’re doing that, think, “Could I just replace this polling system with a call to read()?”

What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more. Whether you are often on the fast path for io_uring will depend on the particulars of your application and its I/O patterns.

Re:Comparison of Rust async and Linux thread context switch ... by scottlamb on Friday February 12, 2021 @12:28AM (#26111012)
> What io_uring does do is provide a way to poll without needing to wait, but if you haven’t received new events when you poll, you’re not on the fast path any more.

Isn't "not on the fast path any more" a bit absolutist? io_uring's "slow" path is roughly one syscall per iteration, right? That's still many fewer syscalls than one syscall per IO operation (or more if any return EAGAIN/EWOULDBLOCK) as you'd be doing without it. I'm not sure I really care about eliminating that last syscall per iteration; it seems minor in comparison.

Re:Comparison of Rust async and Linux thread context switch ... by volta83 on Friday February 12, 2021 @03:21AM (#26111669)
> io_uring's "slow" path is roughly one syscall per iteration, right?

Right, so if a blocking API makes 1 syscall, io_uring would make N syscals for N iterations.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @04:24AM (#26111917)
Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough).

That's pretty huge amortization. Rough benchmarks we've done are showing double throughput for io_uring for 4096 byte AF sector write/fsync/read combos: https://github.com/coilhq/tigerbeetle/tree/master/demos/io_u...

Re:Comparison of Rust async and Linux thread context switch ... by volta83 on Friday February 12, 2021 @07:40AM (#26113004)
> Where 1 iteration and 1 syscall to io_uring_enter() is submitting 100s of I/O operations per io_uring_enter() syscall (and you can even run the ring buffers with the kernel set to poll so you can do 0 syscalls if that's not already enough).

Same for the blocking case. If I do a syscall to read a whole file, its just 1 syscall creating millions of I/O operations.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @10:34AM (#26114755)
Sure, but that's all you'll ever do with the blocking case: 1 syscall at a time, while your program sits and does nothing with the CPU, whereas with io_uring at least you can do CPU while you wait on your IO. So even ignoring the IORING_SQPOLL option that requires no io_uring_enter() syscall, a basic usage of io_uring is still going to be faster.

io_uring is a bicycle for IO, and you can ride it as fast as you want to. But it's apples and oranges to blocking IO, which is always stuck in first gear.

Re:Comparison of Rust async and Linux thread context switch ... by volta83 on Friday February 12, 2021 @11:02AM (#26115068)
> while your program sits and does nothing with the CPU

The CPU can run other threads while the hardware does DMA transfers. The thread just yields when the transfer is started, and a hardware exception wakes it up when the DMA transfer finishes.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @12:22PM (#26116032)
Sure, but we're comparing the efficiency of one of your program's single threads, because otherwise you could take that same argument you just used and turn it around and say fine, just run another thread then with another io_uring... and you're still ahead. You have to compare at the smallest unit of control plane.

At the same time, multiple threads for a single program introduce context switches which are becoming horrendously expensive compared to the sheer number of IOPS that modern NVMe SSDs can do.

Thread-per-core designs built around io_uring are the future of IO on Linux.

Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Friday February 12, 2021 @12:34AM (#26111024)
io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall.

I am a bit baffled how this could possibly be considered an “absolutist” viewpoint—I am just saying that there exist scenarios where io_uring is not helpful. This should be uncontroversial.

Re:Comparison of Rust async and Linux thread context switch ... by jorangreef on Friday February 12, 2021 @04:32AM (#26111944)
> io_uring’s slow path is making one blocking syscall every time you would ordinarily make a blocking syscall.

That's not correct, io_uring was "absolutely" designed, at least in the technical sense, for zero syscalls in the slow path (if you want to):

  IORING_SETUP_SQPOLL
  When this flag is specified, a kernel thread is created to perform submission queue polling.
  An io_uring instance configured in this way enables an application to issue I/O without ever
  context switching into the kernel. By using the submission queue to fill in new submission
  queue entries and watching for completions on the completion queue, the application can submit
  and reap I/Os without doing a single system call.
From the man page: https://manpages.debian.org/unstable/liburing-dev/io_uring_s...

This mode required privileges in early kernel versions but that's already changed. Things are moving fast.

Re:Comparison of Rust async and Linux thread context switch ... by scottlamb on Friday February 12, 2021 @12:47AM (#26111078)
You mean when there's only one thing to do per iteration? I'd describe that as when mostly idle. As the system gets more loaded, the one syscall per iteration matters less and less.
Re:Comparison of Rust async and Linux thread context switch ... by quotemstr on Thursday February 11, 2021 @11:45PM (#26110818)
io_uring is specifically designed so that zero system calls are necessary while the system is busy. Userspace and the kernel both update ring buffers, and ring buffers can be checked and drained without entering the kernel at all.
Re:Comparison of Rust async and Linux thread context switch ... by klodolph on Friday February 12, 2021 @12:03AM (#26110915)
Yes, that’s what “if you haven’t received new events when you poll” means.
Re:Comparison of Rust async and Linux thread context switch ... by the8472 on Friday February 12, 2021 @05:41AM (#26112246)
io_uring doesn't mean async though. You can also use it for blocking batch execution of syscalls. E.g. when you need to stat hundreds of files or wait for several child processes at once. So with some batch-oriented convenience wrappers it can help threaded code too.