Re:Why we use the Linux kernel's TCP stack
by raphaelj on Monday July 11, 2016 @09:48AM
(#12070989)
A single socket can be shared by multiple cores. That means that the kernel must both protect the socket descriptor from concurrent writes, and can't enforce a TCP link to be handled by a defined core (CPU affinity).
[ Parent
| on HN ]
> A single socket can be shared by multiple cores
Absolutely, it can. But everybody sane avoids that, pinning worker processes to specific CPU's and not sharing sockets between them. The rule of thumb is that spinlocks on the hot path of socket access become a bunch of no-ops if there is no lock contention.
I can't see why pinning is such an obvious choice. The kernel's scheduler may decide that CPU 1 should be woken to handle some new packets, because CPU 2 is busy. If you've pinned the socket to CPU 2, you may be losing out.
I get that there are trade-offs between the two modes: pinning can provide better cache usage, you can avoid some locks (but indirectly make the kernel do the work for you) and so on, but I don't see how you can confidently state that pinning is the 'sane' choice.
In an ideal world, the kernel has a better overview of the network state and CPU state, and therefore is best positioned to decide which CPU should handle each packet.
Re:Why we use the Linux kernel's TCP stack
by felixgallo on Monday July 11, 2016 @11:10AM
(#12071567)
The kernel scheduler has no idea which application thread handles which socket. You can formulate an application level plan and then enforce your will with socket and cpu pinning.
Correct, but the difficulty is if your application must share the machine with any other application - even short lived ones. That, I think, is what joosters was alluding to. If the assumption that your application is the only consumer of system resources is broken, then you may see pathological scheduling behavior.