Here's my point of view:
1. Forking is first and foremost a system call. (To be fair, I realize even memory allocation is much of the time, but still.) The kernel is going to do a bunch of work (as fast as it can, of course) and you're going to end up with two different OS level tasks by the end.
2. Those two tasks are now scheduled in tandem by the OS scheduler. For two tasks, this is fine. For hundreds of tasks, it becomes less effective. Threads will rapidly go in and out of I/O wait and the scheduler has to balance all of this.
3. CGI dies right here, though: CGI is not just a fork. It is a fork and exec! The exec, of course, also run in the kernel. It's going to effectively load the binary from scratch. Then you probably hit the linker in usermode, which has to go through the shared library resolution, resolving and mapping shared objects into memory and filling out import tables. This stuff has some caching as far as I know, but still... it's not free.
4. Now you are in the entrypoint of your program... or are you? If you are using CGI with Perl or Python or even Bash, we're not done yet because the script interpreter has to load all of its state from scratch, parse your script, bla bla, and THEN finally we can run your program.
5. Your application now has to do all of its common setup. Every. Connection. If you need to connect to a database, you can't connection pool: you have to open a new socket every time. Redis? same thing. Need to read a config file? Yep, every dang time. You can hack around some of this but in general it's probably going to be like this for most CGI applications. There's a reason why FastCGI exists after all.
The forking model of servers is elegant... but it doesn't work so well in my opinion. The cost of OS-level context switching is non-trivial, forking is relatively expensive, and things get worse when you are talking about something like CGI where your app effectively gets loaded from scratch each time.
My favorite model is definitely the Go model, where the language schedules lightweight threads or fibers across n OS threads (where n = number of logical threads.) It is cheap for the OS scheduler, and the language scheduler can very efficiently deal with things like I/O blocking and GC without huge latency hits.
But you can go about it many ways. Node.JS's event loop model proves pretty effective. Node is far from perfect but I think I would bet on a Node.JS server with a proper event loop over a CGI server forking a C program, myself.
Of course I'm really no expert. But, I think the jump from CGI to FastCGI told me all I need to know: CGI just didn't scale well at all. FastCGI was a much better experience for me, though I no longer use it.
Surprisingly, even thousands of threads can often have better throughput than using event loops and non-blocking IO: https://www.slideshare.net/e456/tyma-paulmultithreaded1
> The cost of OS-level context switching is non-trivial
On modern hardware it's non-zero but trivial: https://eli.thegreenplace.net/2018/measuring-context-switchi...
Hundreds may be a bad example, it's going to vary based on how big your server is but there is a tipping point where threads become very infeasible. I've yet to hit that limit with Goroutines and have literally hit millions of them on relatively small allocations without much of a hitch. Event loops and non blocking IO may perform worse than threads, but I have extreme doubts that threads beat goroutines (and equivalent models like erlang processes.)
Why? Goroutines use the same syscall heavy non-blocking IO as event loops, just with coroutines for programmer convenience.
The biggest difference between Goroutines and pthreads is that Goroutines only have a 2kb default stack size and pthreads have a 2MB stack.
In high thread concurrency situations it's common to turn the pthread stack size down to 48 or 64kb to allow running tens or hundreds of thousands of OS threads. There's even a JVM config flag for it.
2, because the Go scheduler has awareness about the state of the goroutines that an OS scheduler would not, so it can make more intelligent decisions about what goroutines to wake up and when.
You can really have pretty much as many goroutines as you want. Hundreds of thousands of OS threads lands you firmly into tweaking kernel settings land. My local thread max on my desktop is just 127009 - that wouldn't fly for a huge machine running many Go apps, which is exactly the kind of circumstance I was in (using Kubernetes, to be exact.)
This is why thread per request servers like jlhttp are right up there with fasthttp etc in terms of total throughput.
https://www.techempower.com/benchmarks/#section=data-r17&hw=...
Thread pooling is an effective solution to improve webserver performance, and it generally works well. In these synthetic benchmarks, you can't even really see much of a downside. In reality a lot of these benchmarks are only so good because they have requests that complete very quickly, and I think if you add a random sleep() into them many of them will just die outright because they can't handle that much concurrency and block waiting for free workers. You might think that is unrealistic, but consider that many people have Go servers that are making RPCs all over the place, and an actual large amount of time can just be spent waiting on other RPCs. It's a real thing!
And if the world were just responding to HTTP requests, obviously something like Goroutines would be overkill. But, one of my favorite uses of Goroutines was implementing a messaging server where each consumer, queue, exchange were all their own Goroutines. I was inspired by RabbitMQ for the design but unfortunately could not use it in this use case. Luckily Goroutines worked really great here and I was able to scale this thing up hugely. To me this is where they're really great: they're super flexible. They work pretty well for short-lived HTTP requests, but also great for entirely different and more complicated use cases.
Looking back at the benchmark, one of the more interesting approaches here is go-prefork[1], which works by spawning n/2 executables with 2 threads each. I can only imagine the optimal amount of threads was complicated to determine and maybe even has something to do with hyperthreading. Of course the advantage here is the reduced amount of shared state that leads to less contention, and it does indeed show up on the benchmark. In this setup, it looks weird because there's no load balancer (could be something as simple as some iptables rules) or anything in front. In practice, this would be much akin to running separate instances on the same box, which is also a reasonable approach, and I used this approach myself when scheduling servers. Oddly, I don't think they tried the same approach for fasthttp.
I think what else the benchmark shows is how clever you don't have to be in Go to get good performance. go-postgres is routinely in the middle of the pack and it is literally just using the standard library and goroutines in the most basic fashion. It's effectively not optimized. And in reality, in many cases with more complex servers, the overhead is low enough that it isn't worth your time to optimize it much more.
[1]: https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...
Quite frankly, because Go developers don't learn from other's mistakes and reinvent square wheels.
Kernel threads will always be better and faster than usermode threads.
This is because any inefficiency in threading comes from the scheduler. Your usermode scheduler will always necessarily be slower and worse; it just makes so much sense to put your scheduling code where your context switching and process isolation code already is.
Sure, you can get a short-term gain by iterating fast and testing code if kernel developers are too slow and/or unwilling; but then eventually your code will get merged into the kernel anyways once you're done. So really usermode threads are only good as a rapid prototyping tool, not something for serious use.
Your response doesn't even start on the right foot, since Java never used green threads for performance and therefore is not a good place to hedge this argument at. Citation needed if you're going to start with that.
You can also go ahead and insult the Erlang developers for Processes since they did it first:
http://erlang.org/doc/reference_manual/processes.html
>This is because any inefficiency in threading comes from the scheduler. Your usermode scheduler will always necessarily be slower and worse; it just makes so much sense to put your scheduling code where your context switching and process isolation code already is.
You know, a Go context switch doesn't hit the kernel. It's not more expensive than a kernel context switch. I don't know why you'd think it is. Why not inline scheduling to the process where it knows what's blocked on what?
>Sure, you can get a short-term gain by iterating fast and testing code if kernel developers are too slow and/or unwilling; but then eventually your code will get merged into the kernel anyways once you're done. So really usermode threads are only good as a rapid prototyping tool, not something for serious use.
You can hold your breath for threads to become cheaper than goroutines, but careful not to suffocate. A goroutine pretty much just needs a stack, 4kb. The kernel thread has structures, well, in the kernel, for paging, for the task itself, and the stack structure is bigger (think you can at least adjust that though.)
And as for integrating the Go GC into the scheduler... I'd love to see what Linus's thoughts on merging that to kernel are!
Doesn't matter if you just have a few thousand threads but that's a limiting way to look at something. RabbitMQ can have processes all over the place for everything and have more processes than you could ever have threads and it remains among best in class for performance.
Go on the other hand is based on CSP principles and a threading model that looks like actors. The threading model is deeply ingrained in the language, and it is designed around it. Goroutines are very cheap in Go, and the scheduler is fairly effective because it knows what threads to wake when - it's not blind. Goroutines are scheduled across n threads, not just a single OS thread, so they can take good advantage of multicore or multi CPU systems. This design does hurt C interoperability a bit, but imo it's greatly worth it.
Go is not the only language that works this way. I believe its concurrency model was inspired a lot by Erlang with its 'processes' model.