> although most of not all modern libraries have already adapted

Unfortunately kafka, for example, has not: https://github.com/spring-projects/spring-kafka/commit/ae775...

So... What is it seeking to optimize? Why did you need a thread pool before but not any more? What resource was exhausted to prevent you from putting every request on a thread?

It's mainly trying to make you not worry about how many threads you create (and not worry about the caveats that come with optimising how many threads you create, which is something you are very often forced to do).

You can create a thread in your code and not worry whether that thing will then be some day run in a huge loop or receive thousands of requests and therefore spend all your memory on thread overhead. Go and other languages (in Java's ecosystem there's Kotlin for example) employ similar mechanisms to avoid native thread overhead, but you have to think about them. Like, there's tutorial code where everything is nice & simple, and then there's real world code where a lot of it must run in these special constructs that may have little to do with what you saw in those first "Hello, world" samples.

Java's approach tries to erase the difference between virtual and real threads. The programmer should have to employ no special techniques when using virtual threads and should be able to use everything the language has to offer (this isn't true in many languages' virtual/green threads implementations). Old libraries should continue working and perhaps not even be aware they're being run on virtual threads (although, caveats do apply for low level/high performance stuff, see above posts). And libraries that you interact with don't have to care what "model" of green threading you're using or specifically expose "red" and "blue" functions.

You will still have to worry, too many virtual threads will imply too much context switching. However, virtual threads will be always interruptable on I/O, as they are not mapped to actual o.s. threads, but rather simulated by the JVM which will executed a number of instructions for each virtual thread.

This gives the chance to the JVM to use real threads more efficiently, avoiding that threads remain unused while waiting on I/O (e.g. a response from a stream). As soon as the JVM detects that a physical thread is blocked on I/O, a semaphore, a lock or anything, it will reallocate that physical thread to running a new virtual thread. This will reduce latency, context switch time (the switching is done by the JVM that already globally manages the memory of the Java process in its heap) and will avoid or at least largely reduce the chance that a real thread remains allocated but idle as it's blocked on I/O or something else.

What do you mean by context switching?

My understanding is that virtual threads mostly eliminate context switching - for N CPUs JVM creates N platform threads and they run virtual threads as needed. There is no real context switching apart from GC and other JVM internal threads.

A platform thread picking another virtual thread to run after its current virtual thread is blocked on IO is not a context switch, that is an expensive OS-level operation.

Does Java's implementation of virtual threads perform any kind of work stealing when a particular physical thread has no virtual threads to run (e.g. they are all blocked on I/O)?

It does. They get scheduled onto the ForkJoinPool which is a work stealing pool.

"they run virtual threads as needed" - so when one virtual thread is no longer needed and another one is needed, they switch context, yes?

This is called mounting/un-mounting and is much cheaper than a context switch.

This is a type of context switch. You are saying dollars are cheaper than money.

It's been a really long time since I dealt with anything this low level, but in my very limited and ancient experience when people talk about context switching they're talking specifically about the userland process yielding execution back to the kernel so that the processor can be reassigned to a different process/thread. Naively, if the JVM isn't actually yielding control back to the kernel, it has the freedom to do things in a much more lightweight manner than the kernel would have to.

So I think it's meaningful to define what we mean by context switch here.

When a real thread is allocated from a virtual thread to another, the JVM needs to save into the heap the stack of the first virtual thread and restore from the heap the stack of the second virtual thread, see slide 13 of [1]. This is in fact called mounting/unmounting as already pointed out, and occurs via Java Continuation, but from the JVM perspective this is a context switch. It's called JVM and the V stands for Virtual, so yes, it's not the kernel doing it, but it's happening, and it's more frequent the more virtual threads you have in your application.

[1] https://www.eclipsecon.org/sites/default/files/slides/JavaLo...

swapcontext(3) is a userland context switch, and is named so.

The JVM will need to do context switching when reallocating the real thread that is running a blocked virtual thread to the next available virtual thread. It won't be CPU context switching, but context switching happens at the JVM level and represents an effort.

Ok. This JVM-level switching is called mounting/un-mounting of the virtual thread and is supposed to be several orders of magnitude cheaper compared to normal context switch. You should be fine with millions of virtual threads.

It seems that the answer to the question was "memory". Stack allocations, presumably. You have answered by telling us that virtual threads are better than real threads because real threads suck, but you didn't say why they suck or why virtual threads don't suck in the same way.

Real threads don't suck but they pay a price for generality. The kernel doesn't know what software you're going to run, and there's no standards for how that software might use the stack. So the kernel can't optimize by making any assumptions.

Virtual threads are less general than kernel threads. If you use a virtual thread to call out of the JVM you lose their benefits, because the JVM becomes like the kernel and can't make any assumptions about the stack.

But if you are running code controlled by the JVM, then it becomes possible to do optimizations (mostly stack related) that otherwise can't be done, because the GC and the compiler and the threads runtime are all developed together and work together.

Specifically, what HotSpot can do moving stack frames to and from the heap very fast, which interacts better with the GC. For instance if a virtual thread resumes, iterates in a loop and suspends again, then the stack frames are never copied out of the heap onto the kernel stack at all. Hotspot can incrementally "pages" stack frames out of the heap. Additionally, the storage space used for a suspended virtual thread stack is a lot smaller than a suspended kernel stack because a lot of administrative goop doesn't need to be saved at all.

OS Threads do not suck, they're great. But they are expensive to create as they require a syscall, and they're expensive to maintain as they consume quite a bit of memory just to exist, even if you don't need it (due to how they must pre-allocate a stack which apparently is around 2MB initially, and can't be made smaller as in most cases you will need even more, so it would make most cases worse).

Virtual Threads are very fast to create and allocate only the memory needed by the actual call stack, which can be much less than for OS Threads.

Also, blocking code is very simple compared to the equivalent async code. So using blocking code makes your code much easier to follow. Check out examples of reactive frameworks for Java and you will quickly understand why.

> and they're expensive to maintain as they consume quite a bit of memory just to exist, even if you don't need it (due to how they must pre-allocate a stack which apparently is around 2MB initially,

I'm not familiar with windows, but this certainly isn't the case on Linux. It only costs 2mb-8mb of virtual address space, not actual physical memory. And there's no particular reason to believe the JVM can have a list of threads and their states more efficiently than the kernel can.

All you really save is the syscall to create it and some context switching costs as the JVM doesn't need to deal with saving/restoring registers as there's no preemption.

The downside though is you don't have any preemption, which depending on your usage is a really fucking massive downside.

> The downside though is you don't have any preemption, which depending on your usage is a […] massive downside.

Nobody is taking OS threads away, so you can choose to use them when they better fit your use case.

> And there's no particular reason to believe the JVM can have a list of threads and their states more efficiently than the kernel can.

Of course there is. The JVM is able to store the current stack for the Thread efficiently in the pre-allocated heap. Switching execution between Virtual Threads is very cheap. Experiments show you can have millions of VTs, but only a few thousand OS Threads.

I don't know why you think preemption is a big downside?! The JVM only suspends a Thread at safe points and those are points where it knows exactly when to resume. I don't believe there's any downsides at all.

> So... What is it seeking to optimize?

The goal is to maximize the number of tasks you can run concurrently, while imposing on the developers a low cognitive load to write and maintain the code.

> Why did you need a thread pool before but not any more?

You still need a thread pool. Except with virtual threads you are no longer bound to run a single task per thread. This is specially desirable when workloads are IO-bound and will expectedly idle while waiting for external events. If you have a never-ending queue of tasks waiting to run, why should you block a thread consuming that task queue by running a task that stays idle while waiting for something to happen? You're better off starting the task and setting it aside the moment it awaits for something to happen.

> What resource was exhausted to prevent you from putting every request on a thread?

> why should you block a thread

if creating gazillion threads on modern hardware is super cheap why not? I have transparency and debuggability what threads are running, can check stacktrace of each and what are they blocked on.

virtual threads adds lots of magic under the hood, and if there will be some bug or lib in your infra with no vthreads support it is absolutely not clear how to debug it.

Each thread adds overhead.

Some usage types don’t care, some do.

From what I gather virtual threads are an alternative to “callback-hell” (js) or async coloring (python).

> Some usage types don’t care, some do.

I suspect if you care about threads overhead, you won't pick Java, because there will be overhead in other areas too

> From what I gather virtual threads are an alternative to “callback-hell” (js) or async coloring (python).

there is also existing ExecutorService and futures in Java

> there is also existing ExecutorService and futures in Java

Yes, virtual threads are an alternative also to those. (Kind of)

And my frustration is that java had that API for 20 years, it is used everywhere and absolutely battle tested, and now they are adding those virtual threads which break third party libs and make JVM more complicated with various degradations in exchange of benefits most will not notice..

> if creating gazillion threads on modern hardware is super cheap why not?

Virtual threads are a performance improvement over threads, no matter how cheap to create threads are. Virtual threads run on threads. If threads become cheaper to create, so do virtual threads. They are not mutually exclusive.

Virtual threads are on top of that a developer experience improvement. Code is easier to write and maintain.

Virtual threads improve throughput because the moment a task is waiting for anything like IO, the thread is able to service any other task in the queue.

> Virtual threads are on top of that a developer experience improvement. Code is easier to write and maintain.

except now you need to prove somehow that all 100 libs in your project support virtual threads.

> Virtual threads improve throughput because the moment a task is waiting for anything like IO, the thread is able to service any other task in the queue.

from reading similar discussions, linux for example doesn't have true IO async API, you just push lock of Java thread to lock of thread in the kernel

> linux for example doesn't have true IO async API

io_uring has been around for a few years at this point, with vulnerabilities having been fixed to the point that it's fit for broadband usage.

In this discussion people claim io_uring is not real async IO, but just another kernel level threadpool: https://news.ycombinator.com/item?id=38919659

Briefly: The cost of spawning schedulable entities, memory and the time to execution. Virtual threads, i.e., fibers, entertain lightweight stacks. You can spawn as many as you like immediately. Your runtime system won’t go out of memory as easily. In addition, the spawning happens much faster in user space. You’re not creating kernel threads, which is a limited and not cheap resource, whence the pooling you’re comparing it to. With virtual threads you can do thread per request explicitly. It makes most sense for IO-bound tasks.

This article nicely describes the differences between threads and virtual threads: https://www.infoq.com/articles/java-virtual-threads/

I think it’s definitely worth a read.

A thread per request has a high risk of overcommitting on CPU use, leading to a different set of problems. Virtual threads are scheduled on a fixed-size (based on number of cores) underlying (non-virtual) thread pool to avoid this problem.

Why can't virtual threads overcommit CPU use? If I have 4 CPUs and 4000 virtual threads running CPU-bound code, is that not overcommit? A system without overcommit would refuse to create the 5th thread.

I think parent is saying overcommit with OS threads. 4k requests = 4k OS threads. That would lead to the problems parent is talking about.

Why wouldn't 4k virtual threads lead to the same problems?

Because they don't create 4k real threads, and can be scheduled on n=CPU Cores OS threads

4k "real" threads can also be scheduled on 4 CPU cores. What's the difference?

Real threads are extremely expensive both in terms of memory and CPU time compared to virtual threads. I think the main issue is not even that but context switching when switching threads which is also very expensive.

Virtual threads usually require significantly fewer resources to spawn and run. And, if the underlying system is implemented with them in mind, they can use fewer context switches, and possibly even fewer cache misses etc.

The memory overhead of threads.

Just a side note, async JDBC was a thing way before Loom came about, and it failed miserably. I'm not sure why, but my guess would be is that most enterprise software is not web-scale, so JDBC worked well as it was.

Also, all the database vendors provided their drivers implementing the JDBC API - good luck getting Oracle or IBM contribute to R2DBC.. (Actually, I stand corrected: there is an Oracle R2DBC driver now - it was released fairly recently though.)

EDIT: "failed miserably" is maybe too strong - but R2DBC certainly doesn't have the support and acceptance of JDBC.

R2DBC allows to efficiently maintain millions of connections to the database. But what database supports millions of connections? Not postgres for sure, and probably no other conventional database. So using reactive JDBC driver makes little sense, if you're going to use 1000 connections, 1000 threads will do just fine and bring little overhead. Those who use Java, don't care about spending 100 more MB of RAM when their service already eats 60GB.

Reactive drivers were not about 1000 connections, they were about reusing a single connection better, by queuing a little bit more efficient over a single connection. Reactive programming is not about parallelism, it’s about concurrency.

It is not possible to reuse a single connection better, if we're talking about postgres. You must conduct the transaction over a single connection and you cannot mix different transactions simultaneously over a single connection. That's the way postgres wire protocol works. I think that there's some rudimentary async capabilities, but they don't change anything fundamentally.

It might be different for some exotic databases, but I don't see any reason why ordinary JDBC driver couldn't reuse single TCP connection for multiple logical JDBC connections in this case.

It could also be that there just isn’t enough demand for a non-blocking JDBC. For example, Postgresql server is not coping very well with lots of simultaneous connections, due to it’s (a.o.) process-per-connection model. From the client-side (JDBC), a small thread poool would be enough to max out the Postgresql server. And there is almost no benefit of using non-blocking vs a small thread pool.

I would argue the main benefit would be that the threadpool that the developer would create anyway would instead be created by the async database driver, which has more intimate knowledge about the server's capabilities. Maybe it knows the limits to the number of connections, or can do other smart optimizations. In any case, for the developer it would be a more streamlined experience, with less code needed, and better defaults.

I think we’re confusing async and non-blocking? Non-blocking is the part what makes virtual threads more efficient than threads. Async is the programming style; e.g. do things concurrently. Async can be implemented with threads or non-blocking, if the API supports it. I was merely arguing that a non-blocking JDBC has little merit as the connections to a DB are limited. Non-blocking APIs are only beneficial when there are lots, > 10k connections.

JDBC knows nothing about the amount of connections a server can handle, but to try so many connections until it won’t connect any more.

| In any case, for the developer it would be a more streamlined experience, with less code needed, and better defaults.

I agree it would be best not to bother the dev with what is going on under the hood.

Thank you for a A very candid response I enjoyed reading it!

My question is though: Why even do alleged “non-blocking” _at all_? What are people trying to optimize against?

The short answer is that blocking is expensive due to the overhead of the implied context switch and poor locality. As computers become faster, a larger percentage of the CPU time is dedicated to context-switching overhead and non-blocking architectures eliminate that. For applications like databases where this problem is more severe, the difference in throughput between a blocking architecture and a non-blocking architecture can be 10x on the same hardware, so it is a very important optimization if you want your software to have performance that is competitive.

A modern thread-per-core shared-nothing architecture takes this even further and tries to eliminate blocking at the hardware level for the same basic reason.