I might be missing something, but Linux caters to this already:

   SO_RCVLOWAT and SO_SNDLOWAT
          Specify the minimum number of bytes in the buffer until the socket
          layer will pass the data to the protocol (SO_SNDLOWAT) or the user
          on receiving (SO_RCVLOWAT). These two values are initialized to 1.

On the transmit side:

   EPOLLET
      Sets the Edge Triggered behavior for the associated file descriptor.
      The default behavior for epoll is Level Triggered.  See epoll(7) for
      more detailed informa‐ tion about Edge and Level Triggered event
      distribution architectures.

It's also worth noting that kqueue() also doesn't use the socket options for this. Instead, it uses a flag called EVFILT_READ. More here: https://www.freebsd.org/cgi/man.cgi?query=kqueue&sektion=2

Correct. You can set those socket options and change the triggering behavior of epoll. But those socket options do not actually affect how epoll handles the data when it comes in. They are completely ignored by epoll.

Not sure how old your experience is, but I tested from Python via epoll a few minutes after posting (never used that option before!) and it worked as described.

Behind the scenes, Python may have another layer in there to check the socket options, etc, before returning. ie: Just because it looks like it's working on the Python level doesn't mean it's working the same way underneath.

My experience is across several years writing high performance http clients, writing in pure C.

This changed recently so both of you are right in different ways.

Do you have a link to either an announcement of this change, or C code that demonstrates it? If it has changed, I might start doing some things differently...

(2008) https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin...

I dealt with all of this in 2015, and spent three months trying to get epoll to perform similarly to kqueue; namely, to avoid returning on any and every packet received and, preferably, to abide by some sort of minimum bytes received (or connection closing) before returning.

Unless I'm missing something (and I could very well be missing something), I'm not seeing how that is any different than anything I tried...and I went through every socket option I could find, even if it seemed irrelevant.

Have you written code that actually performs this way? Or are you speculating? I really do wish I could do this with epoll, but I (and several others) just never were able to figure it out. And kqueue simply performed at least an order of magnitude better of any variation I could hack together with epoll.

I had neither written code to test it nor was speculating - I was reading the kernel source, which is pretty clear (if you know in general how file polling is implemented in the Linux kernel).

However, because of what you have observed I wrote up a quick test, which confirms to my satisfaction that epoll_wait() respects SO_RCVLOWAT on TCP sockets: https://gist.github.com/keaston/d2473b8b996a34a5860b0744684f...

It seems likely that you must have been dealing with an old "enterprise" kernel in 2015 that hadn't had this fix backported to it?

Wow. When I saw your code, I thought to myself "yeah, but does it work when the socket is for a client?" Turns out, it does.

My only guess is that I was botching something else when I was digging into this a few years ago.

I can't tell you how much I appreciate you putting that together. This seriously changes a lot of what I work on. Thank you.

The client code I wrote: https://gist.github.com/benwills/95ee844853d3b18588fb3df7d56...

Good example.

The errors are just because you need to check for EAGAIN or EWOULDBLOCK when read() returns -1, and not bail out when that happens.

AIO != epoll()

Given that I wasn't asserting that it is, I'm not sure what you're getting at. I was responding to the parent comment.

I guess I misunderstood. It sounded like you were limited by the epoll() interface and felt forced to kqueue() and BSD... but you have other alternatives...