for x in *; do cat "$x" & done | pipeline ...
In the Windows model (in contrast), the programmer says "when this data is available (from this socket, file, etc), put it here (into this memory buffer)" and then waits for the kernel to do something. As a result, you skip that extra syscall -- but you also skip any other read you might have needed to do for data that's arrived at (roughly) the same time. AIO[1] is potentially similar, but it never seems to benchmark well in my experience.
There is no other choice really. Let's say you want to read a specific page from a file. No page is currently in the kernel buffers. If poll where to return non-ready (and remember that poll has no idea which pages you are interested in), what action, within posix, would you take to change it?
poll and friends just do not make sense for plain file descriptors. In linux, splice add an interesting twist, but it is still awkward to use.
pread() is an unusual case. I'd be happy if poll() simply indicated read() would succeed, and I'm happy doing an lseek() before every read().
However a pread() with O_NONBLOCK could return EAGAIN and poll() would then know what pages I'm interested in.
> what action, within posix, would you take to change it?
The general (POSIX-compatible) strategy (as I indicated) has been to fork a thread off to handle reading so that my application is notified when the data is in the fifo. This works for me when I'm reading (say) five files from a slow (or remote) storage, but it's not ideal since it causes a copy.
Also having pread, which is designed for concurrency and scalability, carry state around for poll (an unbounded amount of state btw: you can have multiple threads issuing concurrent preads on the same fd) makes it completely pointless.
There are linux patches floating to add flags to preadv2 so that on a page not present it returns EAGAIN, so that you can optimistically do a sync read an fall back to a thread pool in the slow case. It still doesn't involve poll and friend.
readahead(2) ? posix_fadvise(2) ?
An idea for how it could be implemented in linux: kernel would offer a new kind of file descriptor. It would have a limited, in-kernel buffer, much like a socket has an arrival space. Once you have it, call something like,
newkind_page_copy(newkind_fd, disk_fd, page_count, timeout)
The buffer would behave like socket for the purpose of the read and error lists in select/poll/epoll. If you closed either descriptor, it goes to error-state. Within the kernel, it would watch real_file_fd, using code to similar to whatever is underneath inotify. Like inotify, this would not work for NFS.
? What man page are you reading? The goal of readahead is to prime the cache and return immediately. http://man7.org/linux/man-pages/man2/readahead.2.html
There is a second obstacle I have in mind, around the robustness of this. Imagine if we had an implementation of that which reliably returned quickly. But we take our time doing the subsequent read. By the time we do, the value has already been chased out of the cache. It is an edge-case, and you could probably avoid it in practice, but it would be significant effort to to verify your mechanism, and would create cognitive load.
1. If a read for a range of bytes on a non-blocking file descriptor can be wholly or partly satisfied from the cache, put that data in the read buffer, and return the amount of it
2. If a read on a non-blocking file descriptor can't be satisfied from the cache, return -1, set errno to EAGAIN, and kick off an asynchronous task to load that range of bytes into the cache (an 'asynchronous load')
2a. If there is already an asynchronous load for that range running, don't start another one
3. When an asynchronous load finishes, mark that range as available
1a (surprise!). If that range of bytes was marked as available, remove the mark
4. If there are a nonzero number of available ranges for a file descriptor, return POLLIN from poll()
The way a program with lightweight threads (Node, Go, etc) would use this:
1. Open files as non-blocking
2. Make reads as usual
3. If a read succeeds, or fails other than with EAGAIN, pass the result to the thread and carry on
4. If a read fails with EAGAIN, suspend the thread, and remember the range it was reading
Meanwhile, in a background IO thread ...
5. Repeatedly poll all file descriptors for which there are remembered reads
6. If a file descriptor shows up as POLLIN, run through all the remembered reads for it, and make them again
7. If a read fails with EAGAIN, cool, leave it remembered and carry on
8. If a read succeeds, or fails other than with EAGAIN, pass the result to the thread and resume it
Would that work?
I haven't really thought about partial results of asynchronous reads. If a read asks for two blocks, and one gets loaded, should poll return POLLIN or not? I lean towards yes, because the program can make use of that data. That raises the question of whether to remove the availability mark after the following read. Again, i lean towards yes, because it means asynchronous reads have the same behaviour as ones which partially hit cache. The OS can carry on reading the rest of the range into the cache, but it won't signal the arrival of that data to the program. That should be fine, as the program will probably try to read it soon anyway.
If a program takes too long to get round to reading data once it is available, it might have been pushed out of the cache by then. Too bad. The read will fail with EAGAIN and the whole cycle repeats.
If two threads in a program independently read the same range, only one asynchronous load will be run, and only one thread will end up removing the mark. Programs will need to be careful not to trip up over this; they probably need to coalesce reads on their side of the API.
It sucks to have to retry every outstanding read for a file when it shows POLLIN, but i can't see a way to return information about which ranges are available through the existing calls. A program could keep its list of outstanding reads in time order, scan it from earliest to latest, and stop when it gets a non-EAGAIN result. If there is more than one available range, poll will return POLLIN again and the program will go back and pick it up. If asynchronous loads complete one at a time in order, this reduces the cost to O(1), with a smooth increase in cost as that is less and less the case.
Basically, what i'm saying is that everyone should just implement the kqueue API from BSD.
2. If a read on a non-blocking file descriptor can't be
satisfied from the cache, return -1, set errno to EAGAIN,
and kick off an asynchronous task to load that range of
bytes into the cache (an 'asynchronous load')
If you want pollable file reads, just use eventfd and roll your own. Linux threads are lightweight enough that there's not much reason to care whether they're "kernel" or "user" threads. But pollable file I/O will really only be a win for streaming reads and writes, so you could just use sendfile(), in which case you're reading or writing to a pollable socket; you use a dedicated thread pool to pump data from sockets to the file descriptors, or vice-versa, without any copying.
BTW, Go doesn't need this solution (timeouts notwithstanding) because Go schedules multiple goroutines across multiple kernel-scheduled threads.
If you're really worried about file I/O latency, you should be at least as worried about VM pagefault latency. I disable swap on all my Linux servers. And I also disable overcommit. But my languages of choice (C and Lua) are capable of gracefully handling allocation failure (contrast Python or Go), and I implement my solutions to handle allocation failure without crashing the entire service. Too many other languages make the assumption that memory is infinite. (Alas, large parts of the Linux kernel do this as well, even with overcommit disabled :( Solaris and Windows are the only major OSs I'm familiar with that implement strict and rigorous memory accounting, permitting the system to stay alive (and consistent) under memory exhaustion. FreeBSD might as well, I'm not sure.
https://lwn.net/Articles/219954/
* https://lwn.net/Articles/223899/
* https://lwn.net/Articles/221913/
* https://lwn.net/Articles/316806/
* https://lwn.net/Articles/612038/
* https://lwn.net/Articles/671658/
* http://www.fsl.cs.sunysb.edu/~vass/linux-aio.txt
* http://ogris.de/howtos/splice.html
And so on.
However a few companies have seen real commercial benefit from actually analysing their data, so we're seeing a lot of "big data" interest. Some of these companies have noticed that a bunch of hadoop/mongo/whatever boxes can't even approach the performance of one "big" box tuned right, so we're seeing somewhat of a resurgence in interest here.
Could you expand on that?
(I'm assuming that by the "Windows Way", you mean I/O completion ports.)
I think the main reason Linux doesn't really do it is that asynchronous IO is an intrusive change and there would be just too much work to implement it in all the file systems etc.; i.e. "not worth it".
Many applications which are not OK with synchronous disk I/O seem to find thread pools good enough: reasonably easy to implement, reasonably portable, usually performs ok.
Traditional unix, including Linux pretends to support poll() etc for file I/O (i.e. it supports the interfaces, but the reality is synchronous). The reason for this is that an application that expects async I/O can kinda-sorta-work with real files -- and "kinda sorta" will not be too bad because local disks are fast enough to paper over any problem.
But then if the disks aren't local, why not actually make non-blocking I/O work as advertised?
Actually I can guess the answere: there are 101 corner cases that mean I can't neatly separate out the apps design for noblocking I/O. And that's why I half buy the argument.
But only half.
That's some seriously hairy stuff there. I reckon I'd move to building my app as a kernel module more readily than trying to make a robust async arrangement using that mechanism :) Call me demanding, but I think we should can reasonably expect easier access to async from our OS than this mechanism offers.
On that topic of which.. how's the future of operating systems coming along?
In my opinion, a big issue with these is that asynchronous i/o requests can not be used with select/poll/epoll, but rely on other notification methods instead. AIO can use unix signals (ugh) or thread notification (see sigevent(7)) to signal completion. io_submit uses io_getevents to read events about completed I/O requests.
From a userspace programmer's point of view, it would make much more sense if io_setup() would return a file descriptor that could be used with select/poll, and io_events would be read() from the fd.
Currently it's impossible (?) to have filesystem i/o in the same select/poll loop with your network i/o because of this issue. This is why libuv falls back to using threads and synchronous i/o to provide "async" i/o.
Hacky workaround: don't do filesystem i/o in the first place. Have the kernel serve an iSCSI target, then consume it over the loopback interface entirely in userspace. Your disk now has network semantics! (But you have to bring your own filesystem driver.)
If you don't mind some extra abstraction overhead, you can skip past iSCSI and instead build a userspace NFS client lib into your app, consuming a loopback NFS share. s/NFS/9P/ if your disk server has in-kernel 9P.
(Is there any use to these approaches? Yes: they're actually nearly-optimal ways to persist state from a unikernel running on a hypervisor, because all the disks are remote one way or another anyway. Erlang on Xen uses the 9P approach, for example.)
Still all this event goodness is not good enough for things like infiniband and the brave world of zero copy networking.
You could use a pipe or eventfd or some other fd-based mechanism to wake up a worker thread on a filesystem event, but then you get two context switches (one to wake up io_getevent thread and another to wake up a worker). This carries a lot of indirect costs from jumping back and forth to the kernel.
I am unaware of a solution that would make filesystem i/o and network i/o play nice in one event loop without adding some overhead.
(Windows async disk i/o is thread-backed too behind the scenes, though)
Windows does not spin up or allocate a thread just to wait for an IO operation to complete. Rather, when an IO operation completes, a thread is allocated from the thread pool (hence the name completion ports) on which the callback happens.
So there is no excessive allocation of threads. Everything is event-driven and kernel managed until the operation completes.
People have taken the notion that threads are slow in some circumstances and made the conclusion that threads are slow, period. But that's simply not true.
Those kind of servers are simply quite rare (IME). Nothing wrong with interesting problems in niche spaces, but it's not something you should be worrying about by default.
(No idea what "big" servers provide public stats, but e.g. https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... ) lists a random day for stackoverflow having 209,420,973 http reqests, i.e. a little less than 3000 per second. I doubt context switching is going to matter for them (in the hypothetical world all this was served by one HTTP server, which of course it isn't).
The real cost of threads is that threaded code is far more difficult to get right than non-threaded code. The closest thing I know of to an excuse for using threads is that we are already using threads, so all the complixity is already there.
After all, once every line of code is already a potential race condtion, it hardly matters if you turn a few more of them into real ones.
The only tricky part (API-side) is that write() is defined to confirm that some data got written, or not. If you let write() become non-blocking, and deferred the I/O, perhaps by returning immediately with EAGAIN or some such response, the write is is an unknown state to the rest of the application. What data can then be expected from other read() calls on that file from the same process? The old data? The new? A mix? You'd have to drop the POSIX consistency guarantee. For most programs, this is never likely to be an issue, as they don't tend to be re-reading bits of a file that they are also writing to, so it's a shame there's no option to let the write() be non-blocking if requested.
Issue 2 is failures: If the write, when it eventually happens, fails, you've got to have some way of passing that information back to the process. The solution for this seems easy enough; just let the FD become writeable again and the program will probably re-attempt the same write, at which time you can report the previous error.
Finally, making open() calls non-blocking would be the real API changer. If you defer all of the file opening work and give the application a file descriptor before doing any I/O, then you can hit all kinds of errors (permissions problems, file not found, and so on). How do you pass them back to the program? You can make any future read() or write() calls fail but these syscalls would now have to return error codes that they aren't specified as actually being able to return (read can't return things like EEXIST, EACCESS and so on). So the API is inadequate for these situations. You can see how they could be extended to cover these cases though.