Also, they don't really measure latency correct I think, because they make the mistake of "coordinated omission" (Look it up, Gil Tene, of Zing/Azul fame is its main discoverer).
I was very careful in my wording. You need 10.000 connections, each running 3 req/s for 30.000 req/s. So we are not trying to maximize the system. We are trying to saturate it over a long period with a constant load. And you need to use wrk2 (also from Gil Tene) because otherwise the load generator and the System-under-test ends up coordinating among themselves, hiding the fact that some of the 10.000 connections are starved to get more req/s through on a subset.
The Java frameworks are very very good at handling 512 connections and throwing as many requests through those as possible. What they fail at, is 10.000 connections. What usually happens in bad load generators is that they allow the Java code to starve, say 9500 of those connections while getting all the good numbers on the last 500. wrk2 doesn't allow that to happen, and starvation is counted against the SUT.
As for the source, you would have to wait. I have the raw data, but I have not yet written the blog post. I think I've given you all the information you need to carry out the test if you are so inclined here.
Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.[1]
I bet if you published a test case that proves Java GC impacting the max latency over 5 minutes, that would get the attention of TechEmpower and make a case for extending the test past 15 seconds.
[1] "The high-concurrency plaintext test type is tested at 256, 1,024, 4,096, and 16,384 client-side concurrency." http://frameworkbenchmarks.readthedocs.org/en/latest/Project...
Are you saying that it's common for java web-frameworks to automatically reuse objects and that they do not cause any GC?
> Also, the benchmarks do in fact test all the way up to 16,384 simultaneous connections.
Read jlouis comment once more. It's not that they don't try many connections at once, it's that they take the best result that they get. Big difference.
EDIT: Looking at the code for the Java Netty/JSON test shows that it does not reuse objects. The biggest takeaway from these tests is that it's hard to do performance testing...
Not even close (certainly not when using Quasar). Whether or not you have large GC pauses depends on how you use the heap. If you only allocate objects that live for the duration of the request (which can be enforced by your choice of JVM language) you get the same GC behavior as Erlang (only more general), and a much, much, much, better computation performance, due to HotSpot having one of the world's most advanced optimizing compilers.
However it's worth nothing the test only runs for 15 seconds. It might be an interesting addition to run it for 10 minutes and measure 99.9% latency, as jlouis proposes, and prove one of you correct.
https://www.techempower.com/benchmarks/#section=data-r10&hw=...
That's nonsense. jlouis challenged us to "Take the fastest webserver written for Java and for Erlang", not the slowest ones. And that's exactly what the TechEmpower benchmarks do and they give Java a ~500% advantage.
As for codesize, if you don't like netty then the very next ranked result is servlet3-cass which delivers nearly the same performance -- again, approx 5X faster than the fastest Erlang impl -- in just a few lines of code. Surely you were aware of this when you wrote that comment.
Again if you or anyone can prove a significant GC impact serving up text or Json definitely hit up the techempower folks and make a case for extending the test length. It might help improve Erlang's standing which right now utterly sucks in comparison to Java performance (probably because of the maturity of the JVM).
Yet people seem to think OpenJDK is the only one.
You say BEAM "just works", but it "just works" if you can put up with its non-stellar performance or happen to play directly to its strengths. Any technology just works if you happen to need exactly what it provides. Java and the JVM, however, like the Linux kernel, are a huge project with vast resources, wide reach and very wide applicability from small embedded devices to mainframes. So, naturally, there are a lot of things to choose from, but you are guaranteed to find one that suits your need -- no matter what it is.
Too many Erlang projects start out thinking Erlang is good enough, and then find that they need to write more and more of their code in C. This just cannot happen with the JVM. Not that it's appropriate for everything (it's bad for command-line utilities and Java SE is inappropriate for constrained environments), but it's much harder to miss. You don't need to know or worry about Zing unless you need what it provides (worst-case latency of under 50us); similarly for other tens of thousands of JVM libraries and tools. All you need to know is that you can find a high-quality implementation of whatever you might need down the line.
As for Zing's source code: Zing is proprietary, but as the JVM ecosystem is so big, there's a similar open-source attempt by Red Hat and intended to go into OpenJDK called Shenandoah[1]. Its code is here: http://icedtea.classpath.org/hg/jdk9-shenandoah/hotspot/
Furthermore, Erlang has some things against it in the speed department:
It is forced functional.
It is dynamically typed (A tracing JIT can somewhat alleviate this problem).
But I think the BEAM architecture is pretty sound. It has a lot of things in common with a micro kernel, architecturally. And while micro kernels are not popular currently, they see a lot of use in the embedded space, on sattelite's and so on.
Erlang wins by a large margin because the JVM has to garbage collect and block everyone while doing so. This is not desirable in a soft-realtime system.
I'd definitely not order one as the strict superset of the other.