My argument is that disk I/O is more like memory I/O than it is like network I/O, and so for concurrency purposes it may make more sense to treat it like you would memory I/O (use threads) than like you would network I/O (where you'd use non-blocking APIs and event queues).
It depends on your network and disk - and yes SSD and "slow" ethernet are the common case, but there is enough variation (say an relatively sluggish embedded eMMC on one end and 100 GbE for the networking case), that there's no point in making some distinction about disk vs network latency - for a general concurrency abstraction they're both slow IO and you might as well have a common abstraction like IOCP or io_uring.
> concurrency purposes it may make more sense to treat it like you would memory I/O (use threads) than like you would network I/O (where you'd use non-blocking APIs and event queues).
No, case in point, Windows had IOCP for years such that you could use the same common abstraction for network and disk. The fact that the POSIX/UNIX world was far behind the times in getting its shit together doesn't mean much.
And why, fundamentally, can you not use blocking APIs with threads for networking?
Ah but that's not the distinction I'm making at all. This has nothing to do with the physical latency of the network, which can easily be less than that of a disk. It's about the application-level semantics of your communication.
When you are waiting for a connection to arrive on a listen socket, you have no idea how long you might wait. It might never happen. Someone elsewhere on the network has to take the initiative to connect. Depending on the protocol, the same can be true of waiting for a read or even a write.
> Windows had IOCP
I think Windows is over-engineered in this respect. It doesn't seem to have given Windows any recognized fundamental advantage in running databases.
> And why, fundamentally, can you not use blocking APIs with threads for networking?
Well, you can, especially if you're only talking to a single peer. But when you are talking to multiple parties (especially as part of the same logical task, e.g. talking to multiple back-ends to handle one incoming request) it gets unwieldy to manage a thread for each connection.
Cache is a memory. And which cache, by the way? Even L1 cache on modern processors doesn't have 0 latency. And this is a rather poor way of describing hyperthreading - the CPU doesn't really "switch" - the context for the alternate process is already available and the resource stealing can occur for any kind of stall (including cache loads), not just memory. Calling this a "switch" suggesting it is like a context switch is very misleading. It's not similar conceptually.
In any event, by this definition even a mispredicted branch or a divide becomes "blocking" - which sort of tortures any meaningful definition of blocking.