Reducing the number of send calls, in both the C and Python versions, makes them enormously faster. Go is already batching up the writes, hence the apparent speed advantage.
If you strace the client, you see that the "get" case was replying with two send calls, one for the "VALUE" line, another with the value and "END". All the time is consumed with the client waiting to receive that second message. Depending on the client, and I tried a bunch of ways, it's either in 'poll' (pylibmc), 'futex' (Go), or 'recv' (basic python socket). That second receive is about two orders of magnitude slower than the previous recv.
Why does reading that second line take so much longer?
There's more detail here: https://github.com/grahamking/Key-Value-Polyglot/pull/5#issu...
sock.setsockopt(socket.IPPROTO_TCP, socket.TCP_NODELAY, 1)
It's an interaction between delayed ACKS and the Nagle algorithm, mentioned on the Nagle algorithm wikipedia page.
I'm learning a lot this week. Thanks again.
epoll is optimized for efficiently handling large numbers of sockets, but here there is only one socket. There is no reason epoll should be faster at blocking socket I/O than blocking socket I/O; if it is, I blame the kernel.
(Incidentally, here on OS X where there is no epoll, all the solutions performed pretty terribly - a few seconds for 50000 iterations.)