They're not quite the same kind of thread. But some OS do have support for lightweight threads, e.g. NT. Also OSes today do handle as many threads as most applications ever need.

Java's Virtual Threads (as any other green threads/fibers/etc) are scheduled cooperatively. Effectively a fiber is a tiny data structure which contains a closure which has to be executed next and some metadata.

Virtual threads, or green threads use a co-operative threading/coroutine model on top of the preemptive threads provided by the OS.

In addition to Java virtual threads, similar systems exist in Go goroutines, Rust async and Haskell green threads for example.

There are upsides and downsides to cooperative userspace threads. An erratic or malicious green thread can hog all the CPU time and starve other threads for example.

An operating system can not trust the userspace to behave, starvation due to misbehaving thread is unacceptable.

So the OS can not implement green threads alone, it needs the userspace for the co-operative switching.

In erlang you have preemptive green threads. They don't block* and having a million on one medium server is not unheard of.

* I managed to block a whole erlang VM one day, spoke with Joe Armstrong on chat, but they won't fix this, as it's pretty niche use case.

I'm probably not the only one curious to hear more about this :)

Make a ordered_bag ets table and try to insert several thousand records in one ets call. It will block whole VM (spinning one core at 100%, doesn't matter how many you have) for several seconds. It does this because ordered bag needs to find each key in table as it inserts it, resulting in n^2 complexity and this needs to be done atomically. So for 1 000 keys you have 1 000 000 comparisons during a VM-wide lock. Solution - don't insert so many records in one call into ets table.

I’m more curious to understand how userspace threads can be preemptive. :)

I can think of a few ways:

- The VM just doesn’t JIT and can decide to stop executing a thread by just not interpreting the next piece of bytecode and switching to another green thread instead (this would be pretty slow due to the lack of JIT)

- The VM JITs, but inserts a preamble before every function call saying “Before executing this function, should I switch to another green thread first?”, and thus, so long as you call functions frequently enough, you “preempt” yourself. This is how Go does it, and it’s a well known thing in Go that if you never call a function for a while (like just doing a really huge for loop), the current goroutine doesn’t yield execution and hogs the whole OS thread.

> - The VM JITs, but inserts a preamble before every function call saying “Before executing this function, should I switch to another green thread first?”, and thus, so long as you call functions frequently enough, you “preempt” yourself. This is how Go does it, and it’s a well known thing in Go that if you never call a function for a while (like just doing a really huge for loop), the current goroutine doesn’t yield execution and hogs the whole OS thread.

I'm not sure of the implementation details, but this hasn't been true for a while in Go. As of Go 1.14, goroutines are asynchronously preemptible, so loops without function calls no longer deadlock the scheduler or GC: https://go.dev/doc/go1.14#runtime

The docs there hint at how it’s likely done in go: the runtime uses a signal handler to preempt code that hasn’t yielded the scheduler recently enough.

Interesting! My info is way out of date then. Thanks for the link.

IIRC the Erlang VM does the second (schedule on function call). Since Erlang is a functional language and loops are done via recursion, this works out fine.

Yes, on every function call, also some internal functions implemented in native code are instrumented with those checks. On average thread is switched out after about 2000 "reductions" as they are called. Also there's an optimization, where if you send something to another thread and you are waiting for reply, VM switches instantly to the other thread, which makes some message sending equivalent to a simple function call.

In principle you can also set a timer that raises a signal and then switch to a different coroutine when it fires. But it is a big can of worms and won't perform great.

let's go even further

Why do we even waste resources on OS when we can write e.g UEFI application and run it "directly" and let it manage resources without OS overhead

Nowadays runtimes like Java's, etc, etc. are doing more and more tasks that previously were managed by OSes

Sure, unikernels are a thing.

But being able to run a full OS is convenient, and often still faster overall as there are magnitudes more people working at optimizing general purpose OSs than specialized unikernels.

By the time you got done writing device drivers, etc. you would just have another OS.

https://xkcd.com/927/

Also “without OS overhead” - things sitting in memory don’t necessary detract from performance, but not having hardware acceleration for lack of drivers certainly can.

The only overhead guaranteed during execution by the OS is context switching, and if you want to avoid that you’re basically writing an RTOS for an embedded system.

All else being equal, the one thing Java can do that the OS can’t is avoid paying for context switches. There is a cost to crossing the user space/kernel space boundary that you can avoid with user space lightweight threads.

False. The context will need to be switched regardless of where you're doing it. (Userspace or kernel.)

The cost of crossing the userspace/kernel boundary happens on any system call and has nothing to do with context switching.

(Modern OS's typically context switch during system calls for i/o, exactly the same as userspace threads.)

The page tables change in a kernel context switch, whereas they don't in a user context switch. Depending on your CPU and OS, this may or may not matter; e.g., modern CPUs typically have caches that are tagged with the context (e.g. PCID on Intel), but the Spectre flavor of the day may require that you simply flush the entire L1 cache on context switch anyway if you're unlucky.

The biggest cost of a context switch, though, is typically that your CPU goes to do something else, which reduces the efficiency of any caches (including branch prediction etc.). This is the same whether you have kernel or userspace context switches, assuming the threads you're switching between touch different code and/or data.

Well, page tables don't change on a thread context switch, that is relevant for this discussion.

Why do page tables need to change in a context switch between two threads? AFAIK the kernel is mapped in every process, just not accessible in user mode.

Green threads can be switched between in a single time slice without ever doing a single syscall. This is trivially true: you can make a simple coroutine implementation in C that can switch back and forth between routines thousands of times in a single timeslice.

Technically true, but nobody ever does this because it is a massive waste of CPU cycles.

Both OS and "green" threads are meant to switch when doing i/o.

Userspace IO is a thing though.

Even with traditional kernel side IO, you might not want to attempt a syscall if the socket is known not to be ready.

Also when implementing purely userspace synchronization primitives, being able to swap to a different context purely in userspace is nice.

User/kernel space switch is a different kind of switch. Also when you do it cooperatively, context switches can be a bit more efficient as the runtime has better knowledge about the set of live registers at the switch point.

The "different kind of switch" you talk about doesn't exist on any OS of the past 30 years.

I'm talking about raising a process privilege level, via whatever syscall mechanism. Which is a completely different code path than a process/thread context switch and, among other things, doesn't require going through the scheduler.

Both of course pay something, but virtual threads may pay less.

Is this significant for virtual thread use cases? OS threads are cheap since there's no virtual memory context switch.

The significant difference between virtual and real threads is the stack. Real threads have to support programs written in languages which assume that the stack is a large, contiguous block of memory which they can use without cooperation with the operating system. In practice, that means a large slice of address space has to be allocated for each thread, along with page table entries, and at least a page of physical memory. Whereas virtual threads are deeply integrated with the compiler and runtime which uses them, and don't need to support such programs, meaning they can do more subtle things, with growable stacks, segmented stacks, moving frames off and on the platform thread stack, etc.

I suppose you could implement these more subtle things with real threads, but it would require a richer interface between the program and the operating system, which would need to be supported in compilers and language runtimes. FreeBSD did something like this with kernel scheduled entities, but it never caught on.

As an aside, i wouldn't call virtual threads cooperatively scheduled. To me, that means that user code has to be written with scheduling in mind, to make sure that it hits yield points where the thread can enter the scheduler. Virtual threads don't require that, because the JVM can ensure the necessary entries to the scheduler, since it controls interpretation and compilation. That said, i believe that in the current implementation of virtual threads on the JVM, it is possible that a tight loop that doesn't call any methods will not be rescheduled; i'm hazy about the details of this, because surely this interacts with safepoints. But maybe it's only very slightly cooperatively scheduled!

Just thinking aloud here... So functions already (typically) have a bit of prologue for setting up the stackframe, where they juggle rsp and rbp etc. I wonder if you could change the convention for that into something that basically does RequestStackFrame(SIZE) for some known SIZE. RequestStackFrame could then either grow the stack linearly or get memory in some entirely different region depending on what is needed.

Though doing such a function call for every tiny little function is a bit excessive of course, I'm thinking maybe the compiler should track for every function how much stack it will need in the worst case (usage for the function itself + max of usage for the functions it calls). Functions that use memory below some threshold just grow linearly, while those that use above it will go to the trouble of requesting memory. Now this idea assumes that function calls happen in a DAG, it breaks if functions can call themselves or each other in a cycle. But that's actually a good realization maybe. Like a recursive function should perhaps always explicitly request stack space. Security and robustness benefits!

What about libraries? Ideally library functions would be annotated with how much stack space they need (as a side note I have long thought that C-headers are inadequate for specifying how a function should be called. Not annotating stack space requirements one deficiency here!). Absent such annotations, I suppose we need to go with the status quo of making some guess.

Roughly what you're discussing has been implemented in gcc (behind a flag) for a long time: https://gcc.gnu.org/wiki/SplitStacks

Interesting! It does seem to be missing the part about tracking stack usage at compile time though, unless I'm missing something. The more I think about it, the more that seems like the key part, and dynamically growing the stack being more of a fallback thing.

> That said, i believe that in the current implementation of virtual threads on the JVM, it is possible that a tight loop that doesn't call any methods will not be rescheduled;

Right now virtual threads that don't enter into any blocking code won't be unscheduled by the JVM.

Oh, I thought they could also be scheduled at safepoints? Is that not in there yet, or not going in now?

That's the plan, but it's not there yet.