in what way?

According to my understanding you don't need additional threads for this.

When you read a regular file from disk, and it's not in cache... the thread (or process) calling read() blocks, waiting for the data.

Or perhaps you're using NodeJS-style async callbacks. Guess what happens... Your async read looks like it's just using an event queue, but behind the scenes, NodeJS calls libuv which creates a thread which blocks in read(), and when read() unblocks and returns your callback is added to the event queue.

Perhaps you're using the async/await style with the Promise API for truly async goodness. Guess what happens... Your async call still ends up calling libuv which creates a thread which blocks in read(), and your await waits until libuv's thread read() unblocks and tells the Promise that it's fulfilled.

Maybe you're using shiny io_uring() - finally (it was proposed many times before) a real async system call interface in Linux! Guess what happens... Your lovely async/await calls the pristine goodness of io_uring_enter() which... creates a kernel thread which blocks in read(), and your await waits until the kernel thread's read() unblocks and reports a result.

Even with async/await languages and frameworks, and event-driven frameworks, a blocking thread is used somewhere in the stack to read from a file.

I've measured how many threads to create for optimal throughput of small reads from NVMe storage, and the fastest number of threads varies but it's generally in the tens to low hundreds, and always more than the number of CPU cores, if the data is not already in cache.

This is a complex area to tune adaptively, and Linux itself is unfortunately not ideal for this (it doesn't tell the userspace scheduler about blocked vs preempted I/O threads). Interestingly, reducing the number of CPU cores (using taskset) while keeping the same number of threads sometimes increases I/O throughput.

(Note: The above assumes Linux, not O_DIRECT with libaio, and that data is not in cache when using io_uring).

You seem to be knowledgeable so I'll ask.

[My journey was basically being kool-aided on various green thread variants and async and whatnot, only to realize that threads are pretty damn excellent for most use cases, and all alternatives come with subtle drawbacks.]

Do you think that io_uring (or similar) has a ubiquitous future (as opposed to niche – which there is clearly a demand for atm)? Or is combining kernel and user space scheduling is a forever-hard-to-tune problem? Also, do you see any potential improvements to traditional threading/scheduling that could be helpful alleviating some of the resource hogging issues (which are real but over-hyped imo)? Thanks.

I am not sure about Linux kernel internals and I/O operations. However, on Windows kernel the filesystem (or any driver incl. networking) can complete an I/O operation in non-blocking way, it just marks the IRP (descriptor for I/O operation) is pending and complete it later in different context (i.e. in an interrupt handler even when it is in different process/thread context). Since the user mode application can wait for multiple different kind of objects at the same time (this includes networking, file, mutexes, events), I guess in theory it can really scale when thread_count = core_count.

If a thread is waiting for I/O or a mutex, it will yield and let another thread use that CPU time to do something. This doesn't work if you don't have more threads than logical cores, because then there is no job that can take over.

We've had this type of cpu time sharing far longer than we've had multicore CPUs.

Does this still hold if you use async/await which simply would push a different task on the thread while the other one is waiting?

As far as the operating system is concerned, a thread is a thread. Async/await is a pattern for hiding the implementation details of multitasking or faking it in languages with limited support, it still largely works the same, except you see it as the thread being yielded to rather than from in explicit threading.

Async-await uses non-blocking operations in the kernel. If blocking operations need to occur, then these can occur on a separate thread but this is rare. Most things have a non-blocking equivalent.

Async/await is an abstraction that exists at a layer above OS threads. "Non-blocking" isn't well defined when speaking about OS threading models.