If that is the startup time, are Java AOT compilers taken into consideration into the said benchmarks?
No startup times of course. This is for pushing a simple 2 petabyte workload.
No AOT compilers, just download kafka bin distribution 2.4.1
Just launching 6 or 7 of these.
bin/kafka-run-class.sh org.apache.kafka.tools.ProducerPerformance \
--record-size 1024 \
--topic sfo \
--num-records $((1<<31)) \
--throughput $((1024*64)) \
--producer-props "acks=1" \
"client.id=alex.client" \
bootstrap.servers=172.31.31.28:9092 \
batch.size=81960 \
buffer.memory=$((1024 * 1024)) &> ~/nohup1.txt &Also I started to see a trend in books and blog posts regarding how to write Go code towards better performance, so it isn't a given that it excels at performance out of the box.
All of which comes back to the original point that many times isn't the language, rather how it is written and what tools one makes use of.
If Go is so much better than Java, Google would have replaced it already on Android with Go (battery life and such), instead they went with a mix of AOT/JIT with PGO, introducing Kotlin, while gomobile efforts were never given any serious consideration not even for the NDK.
Go will never replace Java for Android because (1) it doesn’t use a VM, so it would need to compile for every arch that android runs on (2) it would require a bug-compatible port of Android. Because Kotlin runs on the JVM and can run on top of Java codebases, it doesn’t have to leap over these hurdles.
There are other reasons you can come up with I’m sure, but those are the biggest that stick out to me. Notably it has nothing to do with “which language is better”.
They optimize for different use cases, and that’s OK. That being said, if you were writing Android from scratch and were only targeting ARM (an equally silly comparison meant to highlight the differences in the languages), you’d be hard pressed to justify Java over Go.
Google is writing Android from scratch, is it called Fuchsia and Go also doesn't get to play there.
The few parts that were written in Go are scheduled to be rewritten in C++ or Rust, with Dart being the main userspace language.
I entirely admit the possibility, but I see no plan. Hopes and aspirations are not plans.
Instead they went with Rust, C++ and Dart.
Now what have all those languages in common that Go lacks?
This isn't really true for non-trivial code bases. And even if it is, it's not a big difference in practice, especially with incremental compilation where I find it actually much quicker to change a couple of files and re-run unit tests compared to golang which has to spit out a multi-dozen MB binary each time, taking 5-6+ seconds.
> (2) compile into single binaries with no dynamic links
Java is getting AOT compilation which will do the same.
> (3) natively support concurrency and parallelism with M:N routines:threads.
Java is getting those as well: http://cr.openjdk.java.net/~rpressler/loom/loom/sol1_part1.h...
The fact that a tool works out of the box and offers you an extensive array of ways to get more out of it is not in any way worse than a tool that just works out of the box. You can use it in exactly the same way with no more mental effort - stick to the defaults.
What sort of performance measurement uses default configurations? What is even the point of not tuning the GC to your application workload?
I have spent the last 15 years on running Java apps in production and to optimize for the p99 latency is really not that hard. Optimize p99.99999 is a whole different subject though. I don't understand what is the point of comparing a default GC setting that is for hello world applications to a software that is optimized every way possible. Apples to oranges. It is a grat marketing gimmick though. Look ma no performance! Look here, so much faster. We live in a single dimension word, yay!
Seastar is the foundation of Scylla, which shows that rewriting in C++ can deliver magnitudes more performance which is not possible by just tuning Cassandra on the JVM. In fact, Datastax has now copied the Scylla approach in Cassandra but still lags behind drastically with performance.
For any decent SRE out there.
>> magnitudes more performance which is not possible by just tuning Cassandra
Magnitudes?? Are you talking about the order of magnitude? You should read the ScyllaDB performance report first.
https://www.scylladb.com/product/benchmarks/aws-i3-metal-ben...
Avg. 99.9% Latency (ms): 9.9
vs.
Avg. 99.9% Latency (ms): 474.4
While there is no significant latency difference in lover percentile tiers. Where Scylla really shines is TCO. Some companies trade SRE time for license cost, some other companies tune GC.
How is 10ms vs 475 not a major improvement? How is 4 nodes vs 40 not a major improvement? If you're an SRE than how is managing 4 servers with far less tuning and maintenance not a major improvement? Also 99.9% percentile still matters. They're testing with 300k ops/sec which means 300/sec are facing extreme latency spikes that can be enough to fail and/or cause cascading issues through out the application.
There's no metric where Cassandra is better here and you can't tune your way to the same performance in the first place which is the whole point of Scylla. What even is your claim here? Spend more to get less?
What makes it possible to run cassandra/scylla on nodes with TBs of data density is the TWCS compaction strategy from Jeff Jirsa. He was just a cassandra power user at the time, and I like to think that the invention was possible because of Java.
So, next time you read an ad piece from scylla about replacing 40 mid size boxes running CMS with 4 big boxes, don't forget about TWCS.
Scylla is far more than a compaction strategy. If it was that simple, than Cassandra would already be able to do it.
It's an objectively faster database in every metric. Datastax's enterprise distribution has more functionality but core Cassandra is now entirely outclassed by Scylla in speed and features.
The TWCS is just a good example that raw performance is not everything. Performance also comes from things like compaction strategy, data modeling, and access pattern, while users and stakeholders also care about things like easiness to modify, friendly license, and steady stewardship.
Redpanda uses the Seastar framework which was created by the ScyllaDB project. Scylla is high-performance C++ reimplemation of Cassandra and RedPanda seems to be chasing the same thing as an alternative to Kafka/JVM.
As a 12 year adtech veteran who has built ad networks from scratch 3 times, low-latency and high-throughput are critical to ad serving infrastructure and that's why Scylla is such a better alternative to Cassandra. The only other database that gets close is Aerospike, and possibly Redis Enterprise with Flash persistence. It's entirely valid to want similar improvements for event streams as well, and as long as they keep the same external API then you don't lose any of the ecosystem advantages either.
See Gil Tene's talk on how not to measure latency.
https://www.youtube.com/watch?v=lJ8ydIuPFeU
What matters is what most of your customers will get - p99, p999, p9999, p100.
Seastar is a fundamentally different way of programming from what you mentioned above. Let me give you an example. Seastar takes all the memory up front - never gives it back to the operating system (you can control how much via -m2G, etc). This gives you deterministic allocation latency, is just incrementing a couple of pointers. Memory is split evenly across the number of cores and the way you communicate between cores is message passing - which means you explicitly tell which thread is allowed to read which inbox (similar to actors) - i wrote about it here in 2017 https://www.alexgallego.org/concurrency/smf/2017/12/16/futur...
The point of seastar is to not tune the GC for each application workload So to bring that up means that you missed the whole point of seastar. Instead the programmer explicitly reserves memory units for each subsystem - say 30% for the RPC, 20% for the app specific page-cache (since it's all DMA no kernel page cache), 20% for write-behinds, etc. (obviously in practice most of this is dynamic). It is not one dimension as suggested and not apples to oranges. it is apples to apples. You have a service, you connect your clients - unchanged - and one has better latency. that simple.
It may be your experience that when you download a bin kafka say 2.4.1 you change the GC settings but in a multi-tenant environment that's a moving target. Most enterprises I have talked to, just use the default script to startup kafka w.r.t gc memory settings. (they may change some writers settings, caching, etc)
At the end of the day there is no substitute for testing in your own app with your own firewall settings w/ your own hardware. The result should still give you 10x lower latency.
What Scylla is doing is unlocking new performance potential with a C++ rewrite and an entirely different process-per-core architecture that gets around the fundamental limitations of Cassandra and makes it easier to run. This performance and stability has also led to the team making existing C* features like LWT, secondary indexes, and read-repair even faster and better than the original implementations.
What kinds of security assessments are done to guarantee that Scylla is as secure as Cassandra?
W.r.t OSS doing research on it atm.
I had a VP at a major cloud say to me "we will try to use your tech" in person. So doing research before I make the decision
Raft is really easy to parallelize and dispatch to multiple followers async. I measured recently on 3 i3.8xlarge instances which give you 1.2GB/s - and i got around 1.18GB/s sustained -https://twitter.com/emaxerrno/status/1260415381321084929
Also, what's nice about using raft is that if there is a bug, we know it's w/ our implementation and not w/ the protocol. so it gives users sound reasoning.
I consider myself a RAFT expert and worked in kafka for the past 5 years.
Look at stuff like LMAX. Java can be lightning fast.
Not to mention that a lot of libraries that are immediately avail for c++ (see io_uring) take a while to get ported to java. The cost of JNI for a libaio wrapper is also expensive. Last i bench(few years back), switching to a crc32 with JNI switch alone was 30 microseconds - an eternity - before doing the work.
In any case, we use seastar (seastar.io), which I'm not sure can actually be ported to java. The pinned thread per core makes a lot of sense for minimizing latency and cache pollution, etc. Externally, the feeling that apps in java are slow is real, less because JVM is slow per se, but because writing low latency apps in java is not the idiomatic way and those that do see to extract every ounce of performance of the hardware often look else where since the work is just about the same.
If you want to do this, you can gain a lot of performance with having custom allocators and pools on the JVM as well. E.g. frameworks like Netty have pooling strategies for ByteBuffers. If you go that route, you can also gain a lot of performance on the JVM, might really be competitive.
Unfortunately the JVM still enforces too many heap allocations since value-types are not a thing yet, but it still performs well.
One of interesting things I discovered about C++/Rust vs C#/Java is that the former language family wants you to do a lot of optimizations upfront, wich typically results in good performance. But sometimes you also spend too much time into optimizing something something that won't matter in practice.
Whereas the managed languages are a bit easier to work with by default, but will thereby only yield mediocre performance. However you still have the chance to look into the bottlenecks and improve them by large margins using the right approaches. In C# now even more so than in Java thanks to tools like Span and value types.
Perhaps open source engineers are not "paid" enough to spend countless hours carefully optimizing their programs on JVM. Properly paid engineers working on proprietary technology can surely bend and twist JVM and make it perform. It might be possible, but costly due to complexity and unpredictable nature of it.
It reminds me of world of SQL where you have to run your query with many small modifications and hope that the optimizer will generate a sensible query plan while the query is still readable. That's the cost of building on top unpredictable systems. You wonder why you can't simply get access to the lower level - physical plans - and program on that level directly, since you know what your desired outcome on that level is anyway.
I still wonder why new projects for performant data processing are not written for example in Rust since for this task it has all the upsides and none of the downsides of JVM.
If you need really hard limits on collection then that's a tricky problem, but that's also tricky when you're managing memory yourself.
Simple tweaks can go a long way for a lot of developers, but GC performance has been a problem at the last 3 organizations I've been at - and I'm not in the valley or at a FAANG - so it isn't exactly an uncommon scenario for developers.
Unnecessary allocations will be your bottle neck at some point when shutting data around.
10x faster. API compat. No jvm.
I also know of other private impls of it. Just makes sense.