News for hackers. Stuff that matters.
Re:Apache Kafka – Publish-subscribe messaging rethought as a... by teraflop on Wednesday March 12, 2014 @02:29PM (#7387350)
You're right, in principle. However, rewriting it in C++ instead of Java would maybe get you a factor of 2 performance improvement, and that's if it was completely CPU-bound, which isn't a reasonable assumption for a tool designed to manage large disk-resident datasets.

By "hard real-time" I was referring to latency, rather than throughput. Achieving very low latencies is difficult in Java because you don't know exactly when the GC will kick in, but nevertheless it's possible to get very high throughput.

Re:Apache Kafka – Publish-subscribe messaging rethought as a... by MichaelGG on Wednesday March 12, 2014 @02:52PM (#7387484)
But does Java allow easy management of memory, like .NET? From building a moderately high-performance packet-processing system in F#, apart from a few specific encoding parts where the machine code was subpar, the biggest hit to CPU seemed to be the GC. Removing a single allocation from a path provided a measurable improvement. Most of the gain was being able to allocate memory directly (1GB managed heap on top of 16GB+ unmanaged).

With Java, you don't have the unmanaged or struct support, so doesn't that really add up? If you go "native", isn't there significant overhead since you can't have pointers in Java (right - the bytecode doesn't support it?)?

People pull it off, but it seems that GC overhead would be a killer.

Re:Apache Kafka – Publish-subscribe messaging rethought as a... by jorgeortiz85 on Wednesday March 12, 2014 @05:29PM (#7388537)
Kafka uses Linux's zero-copy transfer to move bytes between the network and the disk without going through user-space, let alone the JVM.

There's still GC from objects allocated by Kafka in the JVM, but the actual message data doesn't even go through the JVM.

Re:Apache Kafka – Publish-subscribe messaging rethought as a... by kasey_junk on Wednesday March 12, 2014 @03:10PM (#7387599)
The standard way to do this is to go off heap using the Unsafe packages. Essentially you start handling your own memory, with all that entails.