In most of the comparisons I've seen (usually for Erlang), worst-case latency was the important factor, so interaction with the kernel scheduler was avoided as much as possible. In the Erlang runtime, you can pass a switch to cause the userland scheduler-threads to each get bound to a particular processor core, and to cause the kernel scheduler to avoid scheduling anything on those cores. Effectively, this partitions the processor into a set of cores the OS scheduler entirely manages, and a set of cores that the userland scheduler entirely manages.

It's a lot like a machine running a hypervisor with a single VM on it set to consume 100% of available resources—but actually slightly more efficient than that, since a guest can intelligently pin allocate cores to itself and pin its scheduler-threads to them and then just stop thinking about the pinning, while a hypervisor-host is stuck constantly thinking about whether its vCPUs-to-pCPU mapping is currently optimal, with what's basically a black box consuming those vCPUs.

I know with cgroups in Linux you can pin processes to cores pretty easily. Just curious how Erlang makes this easier.

If you mean that in terms of "does the Erlang runtime intelligently take advantage of the fact that its schedulers are pinned to cores to do things you don't get from plain OS-level pinning", I'm not sure.

I think it might, though. This is my impression from reading, a year or so back, the same docs I just linked; you can read them for yourself and form your own opinion if you think this sounds crazy:

It seems like ERTS (the Erlang runtime: BEAM VM + associated processes like epmd and heart) has a pool of "async IO" threads, separate from the regular scheduler threads, that just get blocking syscalls scheduled onto them. Erlang will, if-and-only-if it knows it has pinned schedulers, attempt to "pair" async IO threads with scheduler threads, so that Erlang processes that cause syscalls schedule those syscalls onto "their" async IO threads, and the completion events can go directly back to the scheduler-thread that should contain the Erlang process that wants to unblock in response to them.†

In the default case, if you don't tell ERTS any different, it'll assume you've got one (UMA) CPU with N cores, and will try to pin async IO threads to the same cores as their paired scheduler-threads. This has context-switching overhead, but not much, since 1. the async IO thread is mostly doing kernel select() polling and racing to sleep, and 2. the two threads are in a producer-consumer relationship, like a Unix pipeline, where both can progress independently without needing to synchronize.

If you want, though, you can further optimize by feeding ERTS a CPU map, describing how the cores in your machine are grouped into CPU packages, and how the CPU packages are further grouped into NUMA memory-access groups. ERTS will then attempt to schedule its async IO threads onto a separate core of the same CPU package, or if not possible, the same NUMA group* as the scheduler-thread, to decrease IPC memory-barrier flushing overhead. (The IPC message is still forced to dump from a given core's cache-lines into the CPU or to NUMA local memory, but it doesn't have to go all the way to main memory.)

ERTS will also, when fed a CPU map, penalize the choice in its scheduling algorithm to move an Erlang process to a different CPU package or NUMA group. (It will still do it, but only if it has no other choice.)

---

† This is in contrast to a runtime without "native" green-threads, like the JVM, where even if you've got an async IO pool, it just sees an opaque pool of runtime threads and sends its completion events to one at random, and then it's the job of a framework like Quasar to take time out of the job of each of its JVM runtime threads to catch those messages and route them to a scheduler running on one of said runtime threads.

The same is true of an HVM hypervisor: without both OS support (paravirtualization) plus core pinning inside each VM, a hardware interrupt will just "arrive at" the same pCPU that asked to be interrupted, even if the vCPU that was scheduled on that pCPU when it made the hypercall is now somewhere else. This is why SR-IOV is so important: it effectively gives VMs their own named channels for hardware to address messages to, so they don't get delayed by misdelivery.

For M:N, there are N kernel stacks.

I don't know of any benchmark comparing Go vs. a 1:1 implementation with 2K pthread stack sizes, but I would be surprised if the performance difference is large at all.

And getting even cheaper than that, with the effort to make kernel stacks use virtual memory. As I understand it, once kernel stacks use virtual memory, they'll start out at a single 4k page.

Doesn't using kernel threads imply lots of context switches? Doesn't that tend to be expensive in terms of time on modern architectures?

No, it's actually fewer context switches. That's because each I/O completion event goes straight from the kernel to the code that was waiting on it (1 context switch), not from the kernel to the userland dispatcher to the code that was waiting on it (2 context switches).

If you're willing to put in a lot of compiler work userland thread switching is a function call, literally. I don't 1:1 threads with small stack are going to compete with that.

Yeah, this doesn't apply for context-switches that happen because of plain old pre-emption, but it does happen if the context-switch is because e.g. a network packet wants to arrive at another process in your VM than the one that's currently running.

I would imagine that there's a reason bare-metal IaaS providers have a business model. :)

Switches occur on timeouts (which involve a round trip through the kernel), on I/O events (which also involve a round trip through the kernel), or on goroutine message passing. So in every case except goroutine message sends, the switches don't actually save you a trip through the kernel.

But ok, for reading you might often have a buffered state of readable while in reality it's not and you only get this information through EAGAIN on read, so you won't avoid the syscall there. For writing you most likely always would.

If you're in that world, you're using C, C++, or That if you're feeling dangerous.

Of course it isn't the OpenJDK, rather specialized JVMs like Azul or they code using C style coding with the GC out of the out paths, having profiled the applications with tools like Java Mission Control.

These are the customers driving Oracles effort for value types, AOT compilation and JNI replacement.

Why not just use C and C++, one might ask?

In spite of all these tricks and required knowledge, salaries are lower than for C and C++ developers and overall project costs are anyway lower due to shorter development times for the other tools not available in C and C++.

Hence why Fintech nowadays is looking into languages like Pony, because they don't want to wait for Java 10, if possible.

So an area where Rust might eventually earn some friends.