Eventually I moved to an async-non-blocking model, and for my initial tests it went great, stuff returned almost immediately and I could grab the next one. The problem is that, sort of by definition, non-blocking stuff doesn't really know when it's going to complete right away, so it just kept enqueuing stuff until it ate all my memory.
I figured out pretty early on what was happening, but the other people on my team really seemed to struggle with the concept of backpressure, I think because they weren't really used to a non-blocking model. You kind of get backpressure for free with blocking code.
Eventually I was able to solve my problem using a BlockingQueue,
The OS scheduler provides no safeguards here (you can use the thread-per-task executor with OS threads - you'll just get an OOME much sooner), and as the virtual thread adoption guide says, you should replace bounded thread pools with a semaphore when needed because it's just a better construct. It allows you to limit concurrency for different resources in an easy and efficient way even when different threads make use of different resources.
In this case, by switching to virtual threads you get to use more appropriate concurrency constructs without needing to manage threads directly, and without sharing them among tasks.
Incoming tasks can be rejected proactively if a system would overflow its limit on concurrent tasks. This approach uses a semaphore’s non-waiting acquire and release operations. See the “bulkhead pattern.”
Alternatively, if a system has a fixed number of threads ingesting tasks, these would respond to backpressure by waiting for available permits before creating more virtual threads.
Is your advice more about things like object pools or connection pools?
However, when you have some background batch computational tasks of some limited concurrency (otherwise you'd be in trouble) then you should have some low-priority platform threads servicing that. Virtual threads work due to Little's law by allowing a high number of threads. For such low-priority background tasks you probably want a very low number of threads (often one), so the ability to have tens of thousands, or millions, of virtual threads won't help you for those background tasks anyway.
Standard concurrency tools like semaphores are capable of reducing the amount of work being done at the same time. You could also simulate classic thread pools by using concurrent lists/sets/etc to replicate traditional work queues, but you'd probably lose any advantage you'd gained from switching to green threads in the first place.
As other have said, this can be achieved with a semaphore and the bulkhead pattern. You can also limit the number of number of connections.
I would expect any "production ready" web server using virtual threads having some sort of limiting. That must be the case right?
The only place where you may want to limit the number of threads is at the entry to the system, but since the semaphore is the construct for limiting the number of anything, you might as well just use that there, too. For example, if the number of requests currently being processed is too high (above the semaphore's number of leases), you don't accept new connections.
1. Managing throughput with virtual threads (it's in Clojure, but it's using the underlying Java APIs)
https://andersmurphy.com/2024/05/06/clojure-managing-through...
The problem is you keep creating async requests, they need to be stored somewhere, and that's in memory. Thread or no thread, it's in memory.
And backpressure is essential, yes. Except for basic tasks.
But there is another solution, which is coalescing requests. I.e. in my current project, I combine thousands of operations, each of which could've been its own async request, into a single "unit of work" which then results in a single async request, for all operations combined.
If we treat IO seriously, we won't just fire off requests into the dark by the hundreds. We'd think about it, and batch things together. This can be done manually, but it's best done at a lower level, so you don't have to think about it at the business logic level.
I haven't done any Java in years, but I always thought OOM errors were recoverable in the JVM. Or is this just a case where the author never thought to catch OOM exceptions? My instinct would be to catch OOM early (i.e. in entry function) in a virtual thread and re-queue the task. In theory re-queueing might fail, too, I guess, but in practice probably not.
This is why I like to code in C (and Lua) for this kind of thing, as in C my core data structures like lists and trees don't require additional allocations for insert. My normal pattern here would be to have, say, three job lists--todo, pending, complete; a job would always exist in one of those lists (presuming it could be allocated and initialized, of course), and no allocations would be required to move a job back from pending to todo on an OOM error during processing.
> My instinct would be to catch OOM early
OutOfMemoryError subclasses VirtualMachineError, and when a VirtualMachineError is thrown, your program seems to be firmly in undefined behavior territory. Quoting the JVM specification [3]:
A Java Virtual Machine implementation throws an object that is an instance of a subclass of the class VirtualMachineError when an internal error or resource limitation prevents it from implementing the semantics described in this chapter. This specification cannot predict where internal errors or resource limitations may be encountered and does not mandate precisely when they can be reported.
[1] https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
[2] https://docs.oracle.com/en/java/javase/21/docs/api/java.base...
[3] https://docs.oracle.com/javase/specs/jvms/se21/html/jvms-6.h...
The reason why these exceptions are so "unsafe" is that they can occur at any moment. Maybe you were in the middle of modifying some data structure which will enter an infinite loop the next time you touch it. It's a bit like signals in C where you have to be extremely careful about the APIs you use, except worse because exceptions unwind the stack. At least in C you can still safely call kernel syscalls regardless of userspace state.
Coming from C, you might think you can just avoid heap allocations for anything that you don't manage with backpressure, but that doesn't work in Java, since Java can only store primitives and references on the stack (at least until project Valhalla is delivered[1]). Almost everything you do triggers a heap allocation.
Virtual Threads make this even worse, since their "stack" is also stored on the heap and it can grow dynamically and require more heap allocations even if you just add a primitive object on the stack!
tl;dr: No, you cannot rely on catching OutOfMemoryError in Java in anything but the simplest scenarios and things break down when you have multiple threads and especially virtual threads.
Can't you use mmap then?
> In a real-world program, you may be doing other things besides downloading URLs and feeding the URLs to the download scheduler. Even though the URL downloader is what's causing the memory pressure, an OutOfMemoryError may very well be thrown by an allocation anywhere else.
But upon that OOM, can't you still release the memory from the downloader so that you have memory for your other allocation?
The one resource you can't easily schedule well is CPU, but that's not a big problem because interactive servers don't work well when the CPU needs to be scheduled (i.e. when it's at 100%). Again, this is true regardless of architecture.
Back when we first released virtual threads, many asked us why we didn't make the default virtual thread scheduler offer time-sharing (with or without priorities). The answer was that we wanted to, but were unable to find workloads for the intended domain of the default scheduler (interactive servers) where any kind of CPU scheduling could help (we said that when somebody finds such real workloads, we'll add fairer CPU scheduling to the default scheduler, assuming there was a good algorithm that would address such workloads).
If you aren't bothering to write a real scheduler then it becomes less clear why you would bother with true async. Lots of hassle for not much payoff.
It's possible we'll start borrowing more patterns from Erlang/OTP since it's been living in a post-greenthreads world for as long as many languages have existed, and has developed patterns around it.
The key phrase in the whole article.
This is the reason I am not fully confident in the "pervasive async" mindset overtaking web development. In JavaScript, pervasive async makes sense, because all functions eventually return control to a central event loop anyway. In environments with real threading, the move from OS threads to "green threads and async everywhere" implicitly gives up the tried-and-tested scheduler of your OS and its mechanisms to handle back-pressure. Developers must now take full responsibility for avoiding buffer bloat.
> they fundamentally change how we think about concurrency limits and resource management.
Virtual threads make concurrency cheaper, but nothing about them (or any other concurrency abstraction) eliminates the need for flow control. By switching to virtual threads, you traded off the safeguards once provided by your (bounded!) thread pool and OS scheduler. It's now your responsibility to put safeguards back in.
As an aside: in Clojure's core.async, channel limits (like 1024 pending puts) are intentionally there as a warning sign that your async pipeline isn't handling back-pressure correctly. This is exactly the "hyperactive downloader with no brakes" problem described in the article. With core.async, you would have eventually hit an exception related to excessive puts on a channel, rather than silently expanding buffers until resource exhaustion.