News for hackers. Stuff that matters.
Re:Open-Sourcing Twitter Heron by fizx on Wednesday May 25, 2016 @03:00PM (#11772217)
Let's say you want a GC'd language (no C, C++) on *nix (no C#), good/varied libraries to work with (no esoteric languages), good performance (no Ruby, Python), reasonable options for implementing concurrency (no Javascript), what's left?

Looks like a JVM language (Java, Scala, etc), or Golang. Java has better tooling and more mature implementations. I personally find modern Java nicer to write than Golang (though Scala nicer than both). These days, it comes down to whether the JVM memory overhead is a big deal on the specific project, and if not, for the class of projects discussed in the previous paragraph, I'm probably choosing Scala (but if my teammates object, then it's back to Java).

Re:Open-Sourcing Twitter Heron by jsmthrowaway on Wednesday May 25, 2016 @03:38PM (#11772470)
> Let's say you want a GC'd language (no C, C++)

IOW, "Let's say you want largely random pauses to impact latency non-deterministically." I'm not knocking GC, but it has serious consequences in systems that make this a weird assumption with which to start, particularly the "big" GC languages that you named. Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were constantly fighting GC pressure (and, indirectly, our achievable concurrency and efficiency) at a household name Scala app. The number of JVM developers who are aware of this and capable with the memory subsystem -- i.e., off-heap strategies such as that used in Flink (relevantly) and some of the clever speedups in netty -- are dwarfed by the number of JVM developers in the whole, so you really need someone who understands these problems to efficiently scale a JVM language.

There are better systems languages if you can move away from GC, such as Rust. And modern C++ is fantastic for systems, and it's too bad it gets overlooked because of bam, your first assumption there. Most systems at Google are in C++ because they invested into the infrastructure that you need to support it. It should tell you something that they're heavily involved in each new version of C++.

(I'm talking about systems, not applications. GC is a viable tradeoff for programmer productivity in applications.)

Re:Open-Sourcing Twitter Heron by snaky on Wednesday May 25, 2016 @05:21PM (#11773224)
> you really need someone who understands these problems to efficiently scale a JVM language

Yes. That's not a problem to find and hire one or two of these guys to write a core. And then hire a ton of regular Java developers to write a ton of code you need around that core, educated and managed by those two smart guys.

Compare it to the problem you have going C, C++, Rust, Nim way. Now you need to hire a ton of guys who can write good C++ code (because it must be good to not to crash every 5 seconds, at least), or even worse - a ton of guys who can write any (not even good) Nim code.

Re:Open-Sourcing Twitter Heron by jsmthrowaway on Wednesday May 25, 2016 @05:31PM (#11773289)
Your entire second paragraph indicates to me that you haven't spent time at a well-oiled C++ shop or potentially haven't worked with modern C++ at all. What does "crash every 5 seconds" even mean? Do you think outside of Java we are all sitting around pounding rocks together and dealing with segfaults all day long? Are you saying JVM languages don't NPE in poor hands?

There are three focus areas of wrongness in your comment (maybe four if I'm feeling particularly culturally trolly) but I'll stick with that one and leave the others to others.

Re:Open-Sourcing Twitter Heron by coredog64 on Wednesday May 25, 2016 @09:19PM (#11774522)
I think he is casting aspersions on the average programmer. I'm on a significant chunk of PRs across projects at my employer and I have seen crazy crazy shit from Java devs with years of experience. In some cases these same devs hold Java certifications from Sun/Oracle. I shudder to think of what would happen if they decided to have these guys write C or C++.
Re:Open-Sourcing Twitter Heron by fizx on Wednesday May 25, 2016 @05:01PM (#11773051)
I'm excited for Rust for the reasons you talk about, but when I tried it a year ago, FFI was broken. I'd like to find a reason to retry it soon!
Re:Open-Sourcing Twitter Heron by steveklabnik on Wednesday May 25, 2016 @06:03PM (#11773495)
Do you remember how it was broken? I wasn't aware of any problems, but if you can remember, I can give you an update.
Re:Open-Sourcing Twitter Heron by jedisfullofit on Thursday May 26, 2016 @08:02PM (#11782789)
"Scala in particular with its immutability generates a lot of garbage if you are not careful, and we were constantly fighting GC pressure (and, indirectly, our achievable concurrency and efficiency) at a household name Scala app. The number of JVM developers who are aware of this and capable with the memory subsystem -- i.e., off-heap strategies such as that used in Flink (relevantly) and some of the clever speedups in netty -- are dwarfed by the number of JVM developers in the whole, so you really need someone who understands these problems to efficiently scale a JVM language."

Jed, once again you are laying claim to experience you do not have. If you ever once had touched Scala at Foursquare it would be a different story. This is just a bunch of tired platitudes.

Re:Open-Sourcing Twitter Heron by PeCaN on Wednesday May 25, 2016 @04:10PM (#11772691)
> what's left?

- Erlang (this fits your criteria at least as Java...)

- OCaml (concurrency isn't the best)

- Haskell (also fits your criteria very well)

- D (easy to use C libraries and sometimes C++)

- Rust (not GCed, but why is GCed a requirement?)

- Mercury (admittedly pretty obscure, but using C libraries is easy enough and it fits everything else)

Java only seems like a good choice if you ignore all the languages that are a better choice than Java. IMO Erlang seems like the choice here (or Elixir if you don't like Erlang syntax).

Re:Open-Sourcing Twitter Heron by mbreese on Wednesday May 25, 2016 @04:28PM (#11772803)
Let's also say that you want to be able to hire programmers to write and maintain it. And it would be nice to make it open to many more people so they could find and fix bugs.

After that, what's left?

At the moment, none of the above languages have any significant talent pool to draw from. If you're operating at "Twitter-scale" (I guess that's now a thing), you also need to be able to spin up a team of people that can get to work quickly without having to learn a language as they go. That eliminates a great number of languages just for logistical reasons. Rust might get there (and I hope it does!), but it's not there yet.

Maybe Twitter already has a significant number of skilled Java engineers, so they decided to use what they already knew really well. There's nothing wrong with that. Choosing any of the other languages you listed would have been a far riskier strategy.

Re:Open-Sourcing Twitter Heron by zjaffee on Wednesday May 25, 2016 @04:39PM (#11772889)
None of these languages even come close to the level of library support and popularity as Java. Additionally it becomes a people problem when you use a language people aren't as familiar with leading to people building connectors which don't perform as well as using the original libraries.

With Java, you can directly import a lot more, especially since the vast majority of OSS in this ecosystem is written in a JVM language.

Re:Open-Sourcing Twitter Heron by hansjorg on Wednesday May 25, 2016 @04:42PM (#11772913)
I think you're underestimating the value of the enormous Java/JVM ecosystem.

That said, I'm hoping to deploy my first Rust project to production soon. It's come a long way for being such a young language.

Re:Open-Sourcing Twitter Heron by PeCaN on Wednesday May 25, 2016 @05:15PM (#11773171)
> I think you're underestimating the value of the enormous Java/JVM ecosystem.

You could be right. For a number of the languages I mentioned I banked on really good FFIs allowing easy usage of C libraries. Perhaps in some domains, including stream processing, there are more and better Java or C++ libraries for them to build on.

Re:Open-Sourcing Twitter Heron by nstart on Thursday May 26, 2016 @01:23AM (#11775692)
Hey.. I'm just super curious to learn a lot about the Rust ecosystem. Would you be up for sharing any details of what your rust project might be? Thanks!
Re:Open-Sourcing Twitter Heron by pron on Wednesday May 25, 2016 @07:01PM (#11773860)
Aside from Erlang which is far too slow (most Erlang apps use C for the data plane) the others are far too risky choices in terms of hiring, future support and general maintainability. Also there are far too many unknowns about them. Rust shows a lot of promise and may very well be suitable for this kind of things, but it may still be too young for an important infrastructure project at Twitter. It is also harder to program than in Java, although for an infrastructure project such as this, this may not be a deal breaker.

Another reason to run on the JVM is that often your applications run on the JVM, and the JIT -- which is getting better and better, and will get incredibly good in Java 9 -- can really do cool stuff when it optimizes across the app/infrastructure line. I've just seen a paper[1] showing 3x performance boost when rewriting parts of SQLite in Python, and letting the JIT optimize the DB together with the app.

[1]: http://arxiv.org/abs/1512.03207

Re:Open-Sourcing Twitter Heron by technion on Friday May 27, 2016 @12:13AM (#11783865)

    Aside from Erlang which is far too slow (most Erlang apps use C for the data plane) 
I'm going to call "citation needed" on that. Riak and EJabberd have a reputation for scale and performance, and the only C I can see referenced is where existing libraries like Zlib get used.

Edit: You are still right. The hiring issues are relevant, and apply to Erlang also.

Re:Open-Sourcing Twitter Heron by hyperpape on Wednesday May 25, 2016 @08:49PM (#11774384)
Is there any kind of overview of the JIT improvements coming in Java 9, or do you have to follow the relevant mailing lists/commits?
Re:Open-Sourcing Twitter Heron by pron on Wednesday May 25, 2016 @09:26PM (#11774568)
I'm referring to Graal, which would be available in Java 9 as an alternative JIT to HotSpot's C2 compiler: https://wiki.openjdk.java.net/display/Graal/Publications+and...
Re:Open-Sourcing Twitter Heron by lpw25 on Friday May 27, 2016 @02:36AM (#11784303)
> OCaml (concurrency isn't the best)

Pet peeve: OCaml's concurrency is pretty good, it's parallelism that it struggles with.

Re:Open-Sourcing Twitter Heron by 010a on Wednesday May 25, 2016 @06:47PM (#11773775)
Finding developers is hard. Finding developers who know those languages to support your software is near impossible.
Re:Open-Sourcing Twitter Heron by abiox on Wednesday May 25, 2016 @10:05PM (#11774719)
java isn't that bad. there's a lot of crappy code-culture that comes from the uber-enterprisey arena, but in more reasonable hands it's quite capable.

it's also worth nothing that Java comes with the JVM and Hotspot JIT, which earn it a bit of swagger.

Re:Open-Sourcing Twitter Heron by flippant on Wednesday May 25, 2016 @04:35PM (#11772853)
- Erlang/Elixir

  - OP mentioned performance. Erlang is terrible for 
CPU-bound tasks.

- Haskell

  - OP said non-esoteric.
- Rust

  - There's a significant ramp up before you learn to deal with the compiler and lifetimes.
Re:Open-Sourcing Twitter Heron by PeCaN on Wednesday May 25, 2016 @05:12PM (#11773147)
> Erlang

Erlang's CPU performance is not bad IME (though admittedly I can't think of anything I've implemented in both Erlang and Java to compare). But that's beside the point; you pick Erlang for networked and I/O bound tasks. Twitter mentioned running Heron on several hundred machines—at that scale, single-thread performance starts to matter less and Erlang's scaling efficiency closes the gap.

> Haskell

Haskell may have been esoteric in 2005, but it's hardly so anymore. (Also, I mention Mercury, and Haskell is the language you decide is too esoteric‽)

> Rust

So you're saying they wouldn't use it because they didn't know it. I suspect that's the real reason they chose Java—not because it is necessarily the best language suited for the task, but because it's the best of the languages Heron's developers knew.

Re:Open-Sourcing Twitter Heron by asdf1234 on Wednesday May 25, 2016 @05:43PM (#11773376)
> Erlang's CPU performance is not bad IME

Erlang is very slow when it comes to CPU bound tasks. It's nowhere near Java for tasks like this.

Re:Open-Sourcing Twitter Heron by galistoca on Wednesday May 25, 2016 @10:06PM (#11774723)
What is the definition of "best" anyway? If you really just think about pure performance nothing can beat a binary code probably. When you say "best" it probably means most productive in the short/long term. In that case it's not about what the language is good at. After all it's humans who use the language. Just like how esperanto is such an awesome concept but nobody uses it and that makes it a useless language, the more people use a language the more valuable it becomes. And yeah Haskell is esoteric. I am confident that I know way many more programming languages than an average programmer but I don't program in Haskell. Try picking 10 random programmers and ask them if they can program in Haskell, at best you'll find 1 (actually that's even being too optimistic)
Re:Open-Sourcing Twitter Heron by insulanian on Wednesday May 25, 2016 @03:57PM (#11772612)
> on nix (no C#)

C# runs absolutely fine on nix.

Re:Open-Sourcing Twitter Heron by int_handler on Wednesday May 25, 2016 @08:18PM (#11774227)
Yes, it is possible to run C# on *nix using Mono but the ecosystem isn't as rich as that of Java.