Re:Why Rust helps even if you have to use a lot of `unsafe`
by RhodesianHunter on Friday August 04, 2023 @04:04PM
(#37005122)
We're not talking about the systems you build on top of here. We're talking about the code. You write yourself. The fact that rocks is written in c is irrelevant to the fact that in order to use the ecosystem I'm describing you need to be on the JVM or your experience will be at best subpar.
Re:Why Rust helps even if you have to use a lot of `unsafe`
by morelisp on Saturday August 05, 2023 @07:09PM
(#37017128)
Sure, if you ignore offloading all the hard work to a C library, we can pretend any language is as fast as any other. It's a boring conversation then though - might as well use Python...
Not to mention you still pay a real performance penalty for being unable to make direct use of RocksDB features, because Kafka is shoving a million "abstraction" and cache layers in there and you just want your your damned merge operator to apply.
I don't agree Kafka off the JVM is subpar. Kafka Streams has a lot of sharp edges from "abstractions" that aren't, Spark is a godawful developer experience and tuning it for performance is black magic for 99% of data engineers, and Flink keeps dithering about how consistent/stable baseline features like queries will be. You can stand up very good apps for a huge set of common cases just as quickly in Go or C++, and not have to deal with Kafka Streams's funny opinions about threads/tasks/consumers.
Then it's still a bit of an untruth since all of those use RocksDB for their high-performance storage layer, which isn't in Java. You could maybe (maybe!) argue for some ergonomics (though especially hard to argue for Spark!) but performance still goes plainly to C (/ C++).