News for hackers. Stuff that matters.
Re:Open-Sourcing Twitter Heron by PeCaN on Wednesday May 25, 2016 @04:10PM (#11772691)
> what's left?

- Erlang (this fits your criteria at least as Java...)

- OCaml (concurrency isn't the best)

- Haskell (also fits your criteria very well)

- D (easy to use C libraries and sometimes C++)

- Rust (not GCed, but why is GCed a requirement?)

- Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else)

Java only seems like a good choice if you ignore all the languages that are a better choice than Java. IMO Erlang seems like the choice here (or Elixir if you don't like Erlang syntax).

Re:Open-Sourcing Twitter Heron by mbreese on Wednesday May 25, 2016 @04:28PM (#11772803)
Let's also say that you want to be able to hire programmers to write and maintain it. And it would be nice to make it open to many more people so they could find and fix bugs.

After that, what's left?

At the moment, none of the above languages have any significant talent pool to draw from. If you're operating at "Twitter-scale" (I guess that's now a thing), you also need to be able to spin up a team of people that can get to work quickly without having to learn a language as they go. That eliminates a great number of languages just for logistical reasons. Rust might get there (and I hope it does!), but it's not there yet.

Maybe Twitter already has a significant number of skilled Java engineers, so they decided to use what they already knew really well. There's nothing wrong with that. Choosing any of the other languages you listed would have been a far riskier strategy.

Re:Open-Sourcing Twitter Heron by zjaffee on Wednesday May 25, 2016 @04:39PM (#11772889)
None of these languages even come close to the level of library support and popularity as Java. Additionally it becomes a people problem when you use a language people aren't as familiar with leading to people building connectors which don't perform as well as using the original libraries.

With Java, you can directly import a lot more, especially since the vast majority of OSS in this ecosystem is written in a JVM language.

Re:Open-Sourcing Twitter Heron by hansjorg on Wednesday May 25, 2016 @04:42PM (#11772913)
I think you're underestimating the value of the enormous Java/JVM ecosystem.

That said, I'm hoping to deploy my first Rust project to production soon. It's come a long way for being such a young language.

Re:Open-Sourcing Twitter Heron by PeCaN on Wednesday May 25, 2016 @05:15PM (#11773171)
> I think you're underestimating the value of the enormous Java/JVM ecosystem.

You could be right. For a number of the languages I mentioned I banked on really good FFIs allowing easy usage of C libraries. Perhaps in some domains, including stream processing, there are more and better Java or C++ libraries for them to build on.

Re:Open-Sourcing Twitter Heron by nstart on Thursday May 26, 2016 @01:23AM (#11775692)
Hey.. I'm just super curious to learn a lot about the Rust ecosystem. Would you be up for sharing any details of what your rust project might be? Thanks!
Re:Open-Sourcing Twitter Heron by pron on Wednesday May 25, 2016 @07:01PM (#11773860)
Aside from Erlang which is far too slow (most Erlang apps use C for the data plane) the others are far too risky choices in terms of hiring, future support and general maintainability. Also there are far too many unknowns about them. Rust shows a lot of promise and may very well be suitable for this kind of things, but it may still be too young for an important infrastructure project at Twitter. It is also harder to program than in Java, although for an infrastructure project such as this, this may not be a deal breaker.

Another reason to run on the JVM is that often your applications run on the JVM, and the JIT -- which is getting better and better, and will get incredibly good in Java 9 -- can really do cool stuff when it optimizes across the app/infrastructure line. I've just seen a paper[1] showing 3x performance boost when rewriting parts of SQLite in Python, and letting the JIT optimize the DB together with the app.

[1]: http://arxiv.org/abs/1512.03207

Re:Open-Sourcing Twitter Heron by technion on Friday May 27, 2016 @12:13AM (#11783865)

    Aside from Erlang which is far too slow (most Erlang apps use C for the data plane) 
I'm going to call "citation needed" on that. Riak and EJabberd have a reputation for scale and performance, and the only C I can see referenced is where existing libraries like Zlib get used.

Edit: You are still right. The hiring issues are relevant, and apply to Erlang also.

Re:Open-Sourcing Twitter Heron by hyperpape on Wednesday May 25, 2016 @08:49PM (#11774384)
Is there any kind of overview of the JIT improvements coming in Java 9, or do you have to follow the relevant mailing lists/commits?
Re:Open-Sourcing Twitter Heron by pron on Wednesday May 25, 2016 @09:26PM (#11774568)
I'm referring to Graal, which would be available in Java 9 as an alternative JIT to HotSpot's C2 compiler: https://wiki.openjdk.java.net/display/Graal/Publications+and...
Re:Open-Sourcing Twitter Heron by lpw25 on Friday May 27, 2016 @02:36AM (#11784303)
> OCaml (concurrency isn't the best)

Pet peeve: OCaml's concurrency is pretty good, it's parallelism that it struggles with.

Re:Open-Sourcing Twitter Heron by 010a on Wednesday May 25, 2016 @06:47PM (#11773775)
Finding developers is hard. Finding developers who know those languages to support your software is near impossible.
Re:Open-Sourcing Twitter Heron by abiox on Wednesday May 25, 2016 @10:05PM (#11774719)
java isn't that bad. there's a lot of crappy code-culture that comes from the uber-enterprisey arena, but in more reasonable hands it's quite capable.

it's also worth nothing that Java comes with the JVM and Hotspot JIT, which earn it a bit of swagger.

Re:Open-Sourcing Twitter Heron by flippant on Wednesday May 25, 2016 @04:35PM (#11772853)
- Erlang/Elixir

  - OP mentioned performance. Erlang is terrible for 
CPU-bound tasks.

- Haskell

  - OP said non-esoteric.
- Rust

  - There's a significant ramp up before you learn to deal with the compiler and lifetimes.
Re:Open-Sourcing Twitter Heron by PeCaN on Wednesday May 25, 2016 @05:12PM (#11773147)
> Erlang

Erlang's CPU performance is not bad IME (though admittedly I can't think of anything I've implemented in both Erlang and Java to compare). But that's beside the point; you pick Erlang for networked and I/O bound tasks. Twitter mentioned running Heron on several hundred machines—at that scale, single-thread performance starts to matter less and Erlang's scaling efficiency closes the gap.

> Haskell

Haskell may have been esoteric in 2005, but it's hardly so anymore. (Also, I mention Mercury, and Haskell is the language you decide is too esoteric‽)

> Rust

So you're saying they wouldn't use it because they didn't know it. I suspect that's the real reason they chose Java—not because it is necessarily the best language suited for the task, but because it's the best of the languages Heron's developers knew.

Re:Open-Sourcing Twitter Heron by asdf1234 on Wednesday May 25, 2016 @05:43PM (#11773376)
> Erlang's CPU performance is not bad IME

Erlang is very slow when it comes to CPU bound tasks. It's nowhere near Java for tasks like this.

Re:Open-Sourcing Twitter Heron by galistoca on Wednesday May 25, 2016 @10:06PM (#11774723)
What is the definition of "best" anyway? If you really just think about pure performance nothing can beat a binary code probably. When you say "best" it probably means most productive in the short/long term. In that case it's not about what the language is good at. After all it's humans who use the language. Just like how esperanto is such an awesome concept but nobody uses it and that makes it a useless language, the more people use a language the more valuable it becomes. And yeah Haskell is esoteric. I am confident that I know way many more programming languages than an average programmer but I don't program in Haskell. Try picking 10 random programmers and ask them if they can program in Haskell, at best you'll find 1 (actually that's even being too optimistic)