e.g. node.js has no problem getting 20k/sec per core, but a stall at the wrong time kills every pipelined HTTP request that follows (until you tear down the connection and restart it).
All of this is motivation to rewrite everything in C.
Reports of "millions per sec" are usually talking about messages on established channels across all CPUs.
If you're actually aware of a java based web server that can beat even 150k http requests measured by wrk or similar on local host I'd like to see it.
I think it can be micro-optimized beyond that. With predictions to avoid unnecessary syscalls, with syscalls grouped together to make cpu more efficient for the rest of the time it spends in event loop, and if it's possible to modify kernel a bit - with batching syscalls together to make them very cheap.
That is to say, I suspect that if micro-optimisations can double our performance, they will be more complicated than just writing a customised ring0 that implements HTTP directly inside the network driver.
Here is how I'm looking at it:
• 10Gb/sec network port
• 4k max requests and responses
• == 1.3 million HTTP requests per second.
Now the problem is that main memory is not much faster than our fastest network: About 15Gb/sec, so what we're talking about here is code and state staying entirely in L1, and streaming the network buffers across the CPU, and responding in one pass, to get that 1.3 million optimal performance.
My dash server gets ~135k HTTP requests per second on localhost (I should be able to approach 300k/sec over a network if I ever get around to it). That's 22% of our optimal performance, and a lot better than any other HTTP server I'm aware of.
At this speed, one of those micro-optimisations `writev()` is actually slower than `write()` -- likely because the code path is shorter in the simpler codebase -- but it illustrates my concern nicely: That we are close to that break-even point with the optimisations we can make. If we make our server bigger and more complicated, it might not make our programs any faster.
That suggests to me that the solution is actually fewer, simpler syscalls, not more, bigger ones.
I took part in a few projects that replaced high throughput servers handling mobile network traffic from C++ to Java.
I work at two ad-tech related companies, one large one medium, both use Java from the beginning. And I know other company using Python/Golang as well.
Didn't know C(not C++) is particular popular until today. Note that ad company are pretty business focused, add or remove features for big clients are pretty common, so development efficiency matters a lot.
> The "stop the world" phase of the collector will almost always be under 10 milliseconds and usually much less.
GC pauses are a killer. They can be worked around, but it takes intensive tuning.
The RTB bidder I wrote many moons ago at a startup was fast as hell, but had problems in the 95th percentile of requests meeting the latency targets, due to GC.
The ad servers at the big ad tech heavy weights are in C++.
Also note I wasn't doing this alone, it was a very big project a mobile operator.
Not that they aren't true, rather their validaty depends a lot of programmer skillset and compilers being used.