As someone who writes code for enterpricey businesses doing a lot of I/O bound stuff, golang style M:N threading is a godsend over Java's standard library, and other common platforms in that space. Being able to express your code in a sequential manor and still gain the performance offered by implicit fiber,goroutine,w/e scheduling is pretty awesome. With futures, there's still some language semantic overhead from the point of view of the developer.
In my problem space, M:N threading is simply worth it. And if I need anything better performing, I can still switch to C for specific use cases.
You don't gain as much performance. On Linux, you don't actually gain that much if anything over 1:1 threading. Most of the benefits of goroutines actually comes from the small stacks, which don't have anything to do with M:N and 1:1 to begin with—they're a feature of GC.
As the blog post states, our end goal is to achieve the ergonomics of Go-style M:N without sacrificing nginx levels of performance. With this futures library, we've established the foundations. It would make no sense to give up before even trying.
> And if I need anything better performing, I can still switch to C for specific use cases.
Why would you write networking code in C in 2016, when there are better alternatives available (like this one)?
Rust's philosophy is to have both performance and ergonomics. It rejects the idea that optimal performance requires a cumbersome programming model. I'm not about to give up on that.
As someone who's done a bit of game development in the past I can easily say that you'll fall flat on your face in some fields if you don't write your own protocols. There are simply no protocols that operate well in every use case and there are also use-cases without standardized protocols.
For extremely high performance operations you NEED to use a custom protocol in some case to squeeze every bit of power out of your applications.
From a game development perspective you need to handle two types of content: content that isn't really ordered by time occurrence and content that needs to happen sequentially.
Because of this you can correctly optimize your networking stack for those needs and get your networking system really fast.
Some people have taken to using TCP with a JSON message passing system and that just doesn't work large scale. You'll have huge latency issues and with a big enough game you have to do some serious bit-packing to make it cost effective to host servers for.
Now I'm speaking generally and this is just from my experience but there is definitely still a reason to write your own network code.
You use green threads instead of native threads because native threads have space overhead, not because they have time overhead. Attempting to spawn 100k OS threads will do strange things to most kernel scheduling algorithms; they're not optimized for that use-case.
The main overhead of a thread, green or native, is the stack. The size of the stack is independent of whether you use native or green threads. Go's small stacks are actually made possible by its GC, not its choice of 1:1 or M:N. In musl, for example, you can have 2KB stacks [1] with 1:1.
I haven't seen a benchmark of huge numbers of native threads vs. a userland scheduler, but I have a hard time imagining that a userland scheduler will beat the kernel's scheduler. The kernel scheduler has a much more global picture of the system compared to userland.
[1]: https://github.com/rofl0r/musl/blob/d05aaedaabd4f5472c233dbb...
In most of the comparisons I've seen (usually for Erlang), worst-case latency was the important factor, so interaction with the kernel scheduler was avoided as much as possible. In the Erlang runtime, you can pass a switch to cause the userland scheduler-threads to each get bound to a particular processor core, and to cause the kernel scheduler to avoid scheduling anything on those cores. Effectively, this partitions the processor into a set of cores the OS scheduler entirely manages, and a set of cores that the userland scheduler entirely manages.
It's a lot like a machine running a hypervisor with a single VM on it set to consume 100% of available resources—but actually slightly more efficient than that, since a guest can intelligently pin allocate cores to itself and pin its scheduler-threads to them and then just stop thinking about the pinning, while a hypervisor-host is stuck constantly thinking about whether its vCPUs-to-pCPU mapping is currently optimal, with what's basically a black box consuming those vCPUs.
I know with cgroups in Linux you can pin processes to cores pretty easily. Just curious how Erlang makes this easier.
If you mean that in terms of "does the Erlang runtime intelligently take advantage of the fact that its schedulers are pinned to cores to do things you don't get from plain OS-level pinning", I'm not sure.
I think it might, though. This is my impression from reading, a year or so back, the same docs I just linked; you can read them for yourself and form your own opinion if you think this sounds crazy:
It seems like ERTS (the Erlang runtime: BEAM VM + associated processes like epmd and heart) has a pool of "async IO" threads, separate from the regular scheduler threads, that just get blocking syscalls scheduled onto them. Erlang will, if-and-only-if it knows it has pinned schedulers, attempt to "pair" async IO threads with scheduler threads, so that Erlang processes that cause syscalls schedule those syscalls onto "their" async IO threads, and the completion events can go directly back to the scheduler-thread that should contain the Erlang process that wants to unblock in response to them.†
In the default case, if you don't tell ERTS any different, it'll assume you've got one (UMA) CPU with N cores, and will try to pin async IO threads to the same cores as their paired scheduler-threads. This has context-switching overhead, but not much, since 1. the async IO thread is mostly doing kernel select() polling and racing to sleep, and 2. the two threads are in a producer-consumer relationship, like a Unix pipeline, where both can progress independently without needing to synchronize.
If you want, though, you can further optimize by feeding ERTS a CPU map, describing how the cores in your machine are grouped into CPU packages, and how the CPU packages are further grouped into NUMA memory-access groups. ERTS will then attempt to schedule its async IO threads onto a separate core of the same CPU package, or if not possible, the same NUMA group* as the scheduler-thread, to decrease IPC memory-barrier flushing overhead. (The IPC message is still forced to dump from a given core's cache-lines into the CPU or to NUMA local memory, but it doesn't have to go all the way to main memory.)
ERTS will also, when fed a CPU map, penalize the choice in its scheduling algorithm to move an Erlang process to a different CPU package or NUMA group. (It will still do it, but only if it has no other choice.)
---
† This is in contrast to a runtime without "native" green-threads, like the JVM, where even if you've got an async IO pool, it just sees an opaque pool of runtime threads and sends its completion events to one at random, and then it's the job of a framework like Quasar to take time out of the job of each of its JVM runtime threads to catch those messages and route them to a scheduler running on one of said runtime threads.
The same is true of an HVM hypervisor: without both OS support (paravirtualization) plus core pinning inside each VM, a hardware interrupt will just "arrive at" the same pCPU that asked to be interrupted, even if the vCPU that was scheduled on that pCPU when it made the hypercall is now somewhere else. This is why SR-IOV is so important: it effectively gives VMs their own named channels for hardware to address messages to, so they don't get delayed by misdelivery.
For M:N, there are N kernel stacks.
I don't know of any benchmark comparing Go vs. a 1:1 implementation with 2K pthread stack sizes, but I would be surprised if the performance difference is large at all.
And getting even cheaper than that, with the effort to make kernel stacks use virtual memory. As I understand it, once kernel stacks use virtual memory, they'll start out at a single 4k page.
Doesn't using kernel threads imply lots of context switches? Doesn't that tend to be expensive in terms of time on modern architectures?
No, it's actually fewer context switches. That's because each I/O completion event goes straight from the kernel to the code that was waiting on it (1 context switch), not from the kernel to the userland dispatcher to the code that was waiting on it (2 context switches).
If you're willing to put in a lot of compiler work userland thread switching is a function call, literally. I don't 1:1 threads with small stack are going to compete with that.
Yeah, this doesn't apply for context-switches that happen because of plain old pre-emption, but it does happen if the context-switch is because e.g. a network packet wants to arrive at another process in your VM than the one that's currently running.
I would imagine that there's a reason bare-metal IaaS providers have a business model. :)
Switches occur on timeouts (which involve a round trip through the kernel), on I/O events (which also involve a round trip through the kernel), or on goroutine message passing. So in every case except goroutine message sends, the switches don't actually save you a trip through the kernel.
But ok, for reading you might often have a buffered state of readable while in reality it's not and you only get this information through EAGAIN on read, so you won't avoid the syscall there. For writing you most likely always would.
If you're in that world, you're using C, C++, or That if you're feeling dangerous.
Of course it isn't the OpenJDK, rather specialized JVMs like Azul or they code using C style coding with the GC out of the out paths, having profiled the applications with tools like Java Mission Control.
These are the customers driving Oracles effort for value types, AOT compilation and JNI replacement.
Why not just use C and C++, one might ask?
In spite of all these tricks and required knowledge, salaries are lower than for C and C++ developers and overall project costs are anyway lower due to shorter development times for the other tools not available in C and C++.
Hence why Fintech nowadays is looking into languages like Pony, because they don't want to wait for Java 10, if possible.
So an area where Rust might eventually earn some friends.
Support for kernel bypass networking libraries like ibverbs, DPDK (has an old unmaintained Rust wrapper) and other IO kernel bypass libraries such as SPDK and IOAT.
If Rust supported these libraries, I'd much prefer the future based Rust code to a massive event loop in C.
It maybe sounds like you should just use Go, which (as you point out) makes exactly those tradeoffs. I think Rust is aiming for a different market segment, with customers that want to make a different set of tradeoffs. Seems ideal to have both options. In particular, I don't think switching to C will ever be something Rust will want to require of its users.
So use go and switch to rust for specific use cases.
Yes! That is basically what I am after.
Currently using Erlang so having to give up cheap isolated processes and message passing to do concurrency would be hard. But I understand that is a different level of abstraction as well (a VM, multi-machine clusters, etc vs almost raw C speed).
But in general bringing Rust and Erlang together seems like a swell idea. Both are focused on safety, which I like. Been keeping my eye on https://github.com/hansihe/Rustler
Would a Scala-style 'for' comprehension achieve the same thing while also being more generally applicable? Scala achieves this by using instance methods 'map', 'flatMap' and 'withFilter'. Personally I think Rust should just lift the sugar wholesale :-)
Is this new approach different because instead of stack frames you just have dynamically allocated callback state?
In the M:N approach you have to allocate stack space for each goroutine that you spawn. This requires that you either know the size of the stack up front (generally not possible without being conservative and requesting a large allocation) or that you start small and grow (resulting in a lot of memory traffic and pauses in the growth case, and much harder to do in C).
By contrast, with the zero-cost futures approach we statically know exactly how much per-goroutine size we will ever need, and we can allocate precisely that amount. Furthermore, we only save the data that's absolutely needed across blocking calls. This results in much smaller per-connection state, and as a result it's quicker to allocate.
It's the difference between static and dynamic control flow. Full M:N requires us to give up static knowledge of what a goroutine will do and try to do the best we can at runtime. With futures, we have a lot more static knowledge, and as a result we can optimize more aggressively.
Well, you are writing the compiler. Sure you'd be up against the halting problem, but relatively few functions are (non-tail) recursive.
Perhaps the unwieldiness of a large stack is better attributed to the feature of unbounded recursion (and FFI into "uncharted territory") than the feature of green threads.
I appreciate that Rust has already been down the road of lightweight threads. This statement just struck me as an assumption that deserved to be questioned.
The futures library is the control flow analysis. Because it uses the type system instead of higher order control flow analysis, it actually achieves precision.
You can actually do a pretty good job without k-CFA.
First, and i know you know this, when you say "fail to produce a bounded stack size", you really mean "a reasonable bounded stack size".
A bounded stack size of easy, and does not require context sensitive analysis: The stack size is just max sum of stack sizes of of meet over all paths in an acyclic graph :)
The cycles, you either can statically calculate the recurrence count or you can't.
Bounded heap size is pretty much impossible, but stack size is pretty easy.
Recursion is also not hard. You form strongly connected components. You try to prove how often the component is cycled. If you can, victory. If you can't, you can bound it unless it's truly dynamic.
More to the point, here's a paper on doing it with ADA in GCC in 2009 (IE with sticks and fire):
http://www.adacore.com/uploads/technical-papers/Stack_Analys...
Note they are within 2% on most cases, and can detect whether it's statically knowable, bounded, or dynamically changing.
But the main point is that code in a high-level programming language looks very different from C or even Rust code. Sparks in Haskell, or processes in Erlang, or even individual goroutines in Go, are typically executing comparatively tiny programs. If the call graph of the program in question is acyclic, then it should be easy to get a bound on the maximum stack size.
To be clear, I agree with you that a 1:1 threading model is the better design for Rust, simply because it's the only design where you can guarantee zero overhead and don't need a complicated runtime. What I don't agree with is that the M:N threading model is inherently inferior. Especially for a high-level programming language, I would assume that you can reduce the overhead significantly through static analysis and a good implementation.
- is not recursive
- does not call function pointers (ie trait objects)
- allocates only fixed-sized objects on the stack
- only calls functions with known stack requirements
then its stack requirement should be known, no? It feels like with these requirements, one can still write many programs (threads, really). And if one goes beyond the restrictions, then they just pay the cost of having to guess a large stack size.
Stack size analysis would be helpful for other applications as well, like embedded platforms.
And a whole program doesn't need to conform, only individual threads. Presumably the ones you want to make a lot of.
(And if a little dynamicism was required, its expense could be paid for at the use, by creating a fresh necessary-sized stack at that point. But I'm probably opening up old split-stack wounds, sorry)
While your general objection is good to always keep in mind, it is a tradeoff. There is no perfect solution when you're up against the halting problem (unless you're proposing to change the language to only bounded recursion).
What I'm arguing for here is for more than the single problem the OP is solving, so it's not a case of choosing one or the other and calling it a day.
Regarding stack vs. dynamically allocated state (in a future) I'm not convinced what's better. Yes, the allocated stack most likely has an overhead. But at least the data will be stored there in linear fashion and accessing the state will be super cheap once the stack is loaded. In case of futures that hold dynamically allocated data the state might be spread over much more memory locations, so it might be slower to access. I however haven't any more scientific data on this.
In my real-world applications I'm getting about the same performance from a Go based and a boost asio (uses no futures but dynamically allocated callback closures for storing state) networking application. However the programming style is completely different.
This makes it comparable to using a stack. In an M:N solution, the stack is usually (but not always) allocated similarly (e.g. using malloc).
Can you quantify the "performance tax associated with M:N"?
When I look at the benchmark of your minihttp vs Go fasthttp, Rust is supposed to be 3x time faster without any GC and yet minihttp pulls 2M req/sec VS 1.5 for Go which is a small margin when you know how Go works and is GC.
I explain in this sibling comment: https://news.ycombinator.com/item?id=12271194
> Do you have any numbers about this implementation?
See the blog post.
I didn't had look into the benchmark but my guess is that it's waiting on the I/O so any language with good concurrent model will have the same numbers.
The benchmark would have been more interesting if it was more CPU intensive, that would have definitely favored Rust.
it's not about Rust Vs Go, it's just that 30% seems very low when you know how fast Rust is vs Go in some scenarios.
Naturally I disagree with all of those items, but I'm also equally uninterested in starting a Rust vs. Go war.
I don't want M:N threading as Go implements it. It's a big loss of performance for marginal benefit over futures. In particular the libmill approach was tried in Rust and the results were far worse than 1:1.
However, assuming this takes off I would like to see async/await style syntactic sugar over futures down the road to make it easier to write code that looks blocking but actually isn't. Crucially, this sugar would maintain the zero-cost nature of futures. With that approach, we'd have the ergonomics of languages like Erlang and Go without the significant performance tax associated with M:N.