News for hackers. Stuff that matters.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pmalynin on Friday February 01, 2019 @11:53AM (#19055797)
After the asymmetric phase, TLS is pretty cheap on modern hardware. About 1500mb/s per core (at least with AES-NI).

This is why all modern Macs do transparent encryption even if you didn’t explicitly turn on FDE.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @01:13PM (#19056719)
Sure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limitations with regards to throughput.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Twirrim on Friday February 01, 2019 @01:42PM (#19057085)
I've worked on services running on java 8 getting high throughput with AES-NI, maxing out 10Gig interfaces. Java is rarely a problem there. Where it is, you can even JNI out to openssl for your AES-NI encryption needs and spare yourself a little CPU power (we discovered that JNI isn't as big a performance hit as people would have you believe)
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @05:38PM (#19059462)
There's a JIRA issue that's been open for three years for kafka to support openssl with JNI:

https://issues.apache.org/jira/browse/KAFKA-2561

Not sure it's going to get done at this point. I've only recently started using kafka and found the TLS issues to be somewhat surprising. AES-NI has been around for how long now?

I speculate that this is because it's come out of the hadoop community and they haven't really had much luck implementing security features.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Twirrim on Saturday February 02, 2019 @12:23PM (#19064231)
https://jaitechwriteups.blogspot.com/2017/10/kafka-with-open...

It looks like with Java 9 the SSL stuff in Kafka saw a drastic speed boost.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pmalynin on Friday February 01, 2019 @01:18PM (#19056782)
I'm not exactly sure what that has to do with the throughput of TLS, especially considering that it's 99% likely that its implemented in a native library (openssl, etc).
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by deanmoriarty on Friday February 01, 2019 @01:22PM (#19056835)
Java TLS implementation is 99.9% implemented in Java itself (look at JSSE), openssl was ditched a long, long time ago.

Not that I would necessarily expect the throughput to be significantly different than the native version considering the good performance of JITted Java, but was just commenting on your assumption.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pmalynin on Friday February 01, 2019 @01:26PM (#19056882)
Oh, I didn't know that, thanks for the clarification.

I mostly work with C++, Python and Node in my day to day, so my assumptions are biased.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by 75dvtwin on Friday February 01, 2019 @11:55PM (#19061177)
Take a look at conscrypt

https://github.com/google/conscrypt

According to reports into Jetty folks, using conscript for TLS, gives 10x improvements

https://webtide.com/conscrypting-native-ssl-for-jetty/

Note, that, from what I read, conscrypt is not compiled for any of the BSDs so if you are testing there, you will have to compile conscrypt yourself

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @01:21PM (#19056822)
The problem is that it's not implemented in the native library. That's why the throughput is so abysmal.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by anotherhue on Friday February 01, 2019 @02:08PM (#19057375)
It also prevents use of the sendfile optimisation -- meaning a lot of Kafka perf disappears.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @05:28PM (#19059388)
One wonders why this was downvoted.