After that, what's left?
At the moment, none of the above languages have any significant talent pool to draw from. If you're operating at "Twitter-scale" (I guess that's now a thing), you also need to be able to spin up a team of people that can get to work quickly without having to learn a language as they go. That eliminates a great number of languages just for logistical reasons. Rust might get there (and I hope it does!), but it's not there yet.
Maybe Twitter already has a significant number of skilled Java engineers, so they decided to use what they already knew really well. There's nothing wrong with that. Choosing any of the other languages you listed would have been a far riskier strategy.
With Java, you can directly import a lot more, especially since the vast majority of OSS in this ecosystem is written in a JVM language.
That said, I'm hoping to deploy my first Rust project to production soon. It's come a long way for being such a young language.
You could be right. For a number of the languages I mentioned I banked on really good FFIs allowing easy usage of C libraries. Perhaps in some domains, including stream processing, there are more and better Java or C++ libraries for them to build on.
Another reason to run on the JVM is that often your applications run on the JVM, and the JIT -- which is getting better and better, and will get incredibly good in Java 9 -- can really do cool stuff when it optimizes across the app/infrastructure line. I've just seen a paper[1] showing 3x performance boost when rewriting parts of SQLite in Python, and letting the JIT optimize the DB together with the app.
Aside from Erlang which is far too slow (most Erlang apps use C for the data plane)
I'm going to call "citation needed" on that. Riak and EJabberd have a reputation for scale and performance, and the only C I can see referenced is where existing libraries like Zlib get used.Edit: You are still right. The hiring issues are relevant, and apply to Erlang also.
Pet peeve: OCaml's concurrency is pretty good, it's parallelism that it struggles with.
it's also worth nothing that Java comes with the JVM and Hotspot JIT, which earn it a bit of swagger.
- OP mentioned performance. Erlang is terrible for
CPU-bound tasks.- Haskell
- OP said non-esoteric.
- Rust - There's a significant ramp up before you learn to deal with the compiler and lifetimes.Erlang's CPU performance is not bad IME (though admittedly I can't think of anything I've implemented in both Erlang and Java to compare). But that's beside the point; you pick Erlang for networked and I/O bound tasks. Twitter mentioned running Heron on several hundred machines—at that scale, single-thread performance starts to matter less and Erlang's scaling efficiency closes the gap.
> Haskell
Haskell may have been esoteric in 2005, but it's hardly so anymore. (Also, I mention Mercury, and Haskell is the language you decide is too esoteric‽)
> Rust
So you're saying they wouldn't use it because they didn't know it. I suspect that's the real reason they chose Java—not because it is necessarily the best language suited for the task, but because it's the best of the languages Heron's developers knew.
Erlang is very slow when it comes to CPU bound tasks. It's nowhere near Java for tasks like this.
- Erlang (this fits your criteria at least as Java...)
- OCaml (concurrency isn't the best)
- Haskell (also fits your criteria very well)
- D (easy to use C libraries and sometimes C++)
- Rust (not GCed, but why is GCed a requirement?)
- Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else)
Java only seems like a good choice if you ignore all the languages that are a better choice than Java. IMO Erlang seems like the choice here (or Elixir if you don't like Erlang syntax).