News for hackers. Stuff that matters.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by anotherhue on Friday February 01, 2019 @10:59AM (#19055134)
50MB/s is tiny, you can push 500 on commodity disks.

https://www.slideshare.net/ConfluentInc/kafka-on-zfs-better-...

Full talk: https://www.confluent.io/kafka-summit-sf18/kafka-on-zfs (shameless plug)

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by andmarios on Friday February 01, 2019 @01:36PM (#19056998)
I think it's a good number for a sustained load within a large cluster.

50MB/sec multiplies due to replication and consumers. Also in large clusters, the bottleneck comes often from the network switches rather than the rest of the stack.

Also when it comes to sustained load, many people don't realize that 50MB/sec × 10 brokers × 24 hours × 7 days × 3600 seconds leads to 300TB per week (without counting the replication). :)

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by zamadatix on Saturday February 02, 2019 @06:57PM (#19066731)
32 port 100G switches are dirt cheap. If the network is your bottleneck you likely overspent on the wrong hardware.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vaishaksuresh on Friday February 01, 2019 @12:35PM (#19056288)
Agreed, we regularly do 500+ MB/s bytes out on 12 broker cluster and we've tested more than twice that. We've seen brokers handle close to 5GB/s before performance degraded.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @01:21PM (#19056810)
I understand that you can increase single-broker throughput, but one wonders what use case requires that? That'd be warranted in the case of single-partition topics which are necessary to preserve strict ordering. If one has ability to create enough partitions to mitigate hot spots, I'm not sure I see the need.

My org has gone a different direction. We're using spinning rust in GCP and deploying on kubernetes to make scaling out easy. Standard disk provides surprisingly good performance (relative to AWS) and we can easily ingest 10gbps with 20-30 brokers.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by anotherhue on Friday February 01, 2019 @02:07PM (#19057365)
This was a three node cluster, with 3 replicas.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @05:41PM (#19059475)
My question, perhaps poorly articulated was why would someone focus on improving single-node performance vs. cluster-wide performance? I wasn't implying that one would want to run a single-node cluster.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @11:50AM (#19055755)
Yeah, I've got a Ceph cluster in my homelab that tops out about 80mb/s and it's pretty darn painful compared to a 2/3 disk mirrored ZFS on spinning rust.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by scurvy on Friday February 01, 2019 @04:14PM (#19058692)
You're doing something very wrong if that's what Ceph maxes out at. It's almost like you're trying to do this on a 1gig network.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @04:38PM (#19058916)
Well it is a homelab, if you want to front me the cost of a 10Gbe switch be my guest ;).

Even then though I can get 125mb/s throughput from iperf, it's more just pointing out that Ceph has some pretty hefty bandwidth overhead. No knock against Ceph, it's a cool piece of tech and has its place if you've got the pipes to support it. The comment was more about how impressive ZFS is.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by scurvy on Friday February 01, 2019 @07:40PM (#19060167)
10gig stuff is pretty dirt cheap these days. I wouldn't mess around with 1gig and draw any conclusions about anything.

125MB/sec with iPerf is the max throughput of a 1gig link. Then again, iPerf doesn't do anything with the data. Ceph does a lot with the data once it's pulled off the network (mainly replication or erasure coding to other OSD's).

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @08:47PM (#19060463)
Eeeeh. They're getting reasonable in price but I wouldn't call them dirt cheap. Since it's a homelab that lives in my house that rules out anything that sounds like a jet engine. If I match my other ubnt gear you're looking $400+ for a SFP+ capable switch.

I can hit 125MB/s on my ZFS migrations so it's not a unachievable number, just that Ceph had overhead(which is fine, it's doing something different than ZFS).

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by scurvy on Friday February 01, 2019 @09:45PM (#19060679)
You can find used 40gig switches that break out to 10gig ports at less than $4 per port. That's pretty cheap.

Even the older Arista and whitebox QCT stuff is less than $200-250 per switch. For a lab, I'd rather use something that once lived in a datacenter than something consumer-facing like Ubiquiti.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @10:09PM (#19060777)
Well that's the beauty of a homelab, I can run whatever I want in it. Since it's all just for experimentation I don't see an issue running 1gbe. For what it's worth I did give a 3x bonded lcap link a try but since Ceph uses a single connection per node it doesn't scale across the aggregated link.

However at the end of the day if I don't want to keep the kid up at night I gotta run gear that doesn't sound like an F-18 at full throttle.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by polskibus on Friday February 01, 2019 @01:01PM (#19056591)
Are there any good ceph alternatives? Why can't you use zfs directly?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vvanders on Friday February 01, 2019 @03:36PM (#19058296)
Yeah, there's a few but I haven't had a chance to give them a spin yet. I was mostly using Ceph to explore live migrations in Proxmox, although they just released support for ZFS based migrations(you have to migrate the whole local disk though so they aren't nearly as snappy).

There's some pros to Ceph, mostly that you can add/remove drives at-will and the cluster will automatically rebalance. It's just very, very bandwidth hungry for cross-cluster communication.

It doesn't matter that much since it's for a homelab, I could migrate to some 10Gbe links between the cluster at some point and it'll get snappier. ZFS is just such a solid, battle tested piece of tech that it's hard to beat it unless you have a very different set of constraints.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by maccam94 on Friday February 01, 2019 @05:23PM (#19059349)
Ceph is distributed storage, ZFS is direct-attached to one host at a time.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by dikei on Saturday February 02, 2019 @02:41AM (#19061733)
You can't really compare the two number, the Dropbox's post doesn't mention how many disks they are using for each brokers.

In your example, they need 16 HDDs to reach 500MB/s, which average to ~30MB per disk.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @11:27AM (#19055486)
With TLS?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pmalynin on Friday February 01, 2019 @11:53AM (#19055797)
After the asymmetric phase, TLS is pretty cheap on modern hardware. About 1500mb/s per core (at least with AES-NI).

This is why all modern Macs do transparent encryption even if you didn’t explicitly turn on FDE.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @01:13PM (#19056719)
Sure, sure, but did you get that kind of throughput with kafka on java? Most folks (Confluent, for example) are still on java 8 which has some very severe limitations with regards to throughput.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Twirrim on Friday February 01, 2019 @01:42PM (#19057085)
I've worked on services running on java 8 getting high throughput with AES-NI, maxing out 10Gig interfaces. Java is rarely a problem there. Where it is, you can even JNI out to openssl for your AES-NI encryption needs and spare yourself a little CPU power (we discovered that JNI isn't as big a performance hit as people would have you believe)
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @05:38PM (#19059462)
There's a JIRA issue that's been open for three years for kafka to support openssl with JNI:

https://issues.apache.org/jira/browse/KAFKA-2561

Not sure it's going to get done at this point. I've only recently started using kafka and found the TLS issues to be somewhat surprising. AES-NI has been around for how long now?

I speculate that this is because it's come out of the hadoop community and they haven't really had much luck implementing security features.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Twirrim on Saturday February 02, 2019 @12:23PM (#19064231)
https://jaitechwriteups.blogspot.com/2017/10/kafka-with-open...

It looks like with Java 9 the SSL stuff in Kafka saw a drastic speed boost.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pmalynin on Friday February 01, 2019 @01:18PM (#19056782)
I'm not exactly sure what that has to do with the throughput of TLS, especially considering that it's 99% likely that its implemented in a native library (openssl, etc).
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by deanmoriarty on Friday February 01, 2019 @01:22PM (#19056835)
Java TLS implementation is 99.9% implemented in Java itself (look at JSSE), openssl was ditched a long, long time ago.

Not that I would necessarily expect the throughput to be significantly different than the native version considering the good performance of JITted Java, but was just commenting on your assumption.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pmalynin on Friday February 01, 2019 @01:26PM (#19056882)
Oh, I didn't know that, thanks for the clarification.

I mostly work with C++, Python and Node in my day to day, so my assumptions are biased.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by 75dvtwin on Friday February 01, 2019 @11:55PM (#19061177)
Take a look at conscrypt

https://github.com/google/conscrypt

According to reports into Jetty folks, using conscript for TLS, gives 10x improvements

https://webtide.com/conscrypting-native-ssl-for-jetty/

Note, that, from what I read, conscrypt is not compiled for any of the BSDs so if you are testing there, you will have to compile conscrypt yourself

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @01:21PM (#19056822)
The problem is that it's not implemented in the native library. That's why the throughput is so abysmal.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by anotherhue on Friday February 01, 2019 @02:08PM (#19057375)
It also prevents use of the sendfile optimisation -- meaning a lot of Kafka perf disappears.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @05:28PM (#19059388)
One wonders why this was downvoted.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pwaivers on Friday February 01, 2019 @10:44AM (#19054988)
The overload indicators are strange to me. For example, one indicator is "IO thread idle below 20%: this means the pool of worker threads used by kafka for handling client requests are too busy to handle any more workload". How did they determine this? Is it true?

I think a much better way would be to monitor the JMX reporter for dropped messages or high latency: https://docs.confluent.io/current/kafka/monitoring.html.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pram on Friday February 01, 2019 @03:03PM (#19057933)
There are JMX beans for the socket server and request handler idle. If they even dip below 80% you’re probably looking at poorly configured producers timing out.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pandemic_region on Friday February 01, 2019 @10:48AM (#19055022)
Line is doing 150 billion msgs/day on their Kafka infra, about 3million per second. https://www.slideshare.net/linecorp/building-a-companywide-d...
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by buro9 on Friday February 01, 2019 @11:24AM (#19055461)
At Cloudflare a year ago we said we were doing 6m per second ingest rate on a cluster of 106 brokers with x3 replication factor, 106 partitions: https://blog.cloudflare.com/http-analytics-for-6m-requests-p...

The more interesting blog post IMHO is this one on Kafka compression https://blog.cloudflare.com/squeezing-the-firehose/

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Rebelgecko on Friday February 01, 2019 @12:23PM (#19056136)
The compression article was really useful for me when I was trying to squeeze a bit more performance out of our brokers at work. Any plans to revisit the article now that Kafka has zstd support?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by nosequel on Friday February 01, 2019 @12:58PM (#19056549)
They talk about zstd in the post and they were the ones who added support to Sarama.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @06:39PM (#19059861)
I'm using zstd with my kafka implementation and it works great. v2.1 is required. We're using librdkafka, which works, but I believe we had to compile it in to get that and idempotent producer.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by tudelo on Friday February 01, 2019 @11:21AM (#19055421)
That seems... like a lot. Thanks for the link.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by shanemhansen on Friday February 01, 2019 @01:56PM (#19057239)
I regularly do 200MB/s per node with NSQ (I know, I know, apples to oranges but people should be aware there are choices other than apples depending on your use case).
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Rafuino on Friday February 01, 2019 @12:56PM (#19056510)
60 MB/s per broker at what latency though? Does latency even factor into the performance consideration?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @01:23PM (#19056841)
For some use cases, it absolutely does. I can't remember the exact use case, but a friend of mine is adopting kafka with an end-to-end latency goal of 100ms. Absolutely requires SSDs and for the producers and consumers to be colocated, but is otherwise not that difficult to achieve.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by hsong on Saturday February 02, 2019 @12:06AM (#19061221)
It would be great to see analysis on where the bottleneck is. Is it disk, network, compression, etc.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Zecar on Friday February 01, 2019 @01:09PM (#19056677)
Dropbox deleted my honeymoon photos
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by deanCommie on Friday February 01, 2019 @10:33AM (#19054894)
"At Dropbox, Kafka clusters are managed by the Jetstream team, whose primary responsibility is to provide high quality Kafka services. "

And this is why AWS's "Managed Kafka" service has a place.

I get why Dropbox moved off S3 - storage is their core competency and they thought they could/should do better/cheaper. But I'm surprised they would be wasting valuable dev time with this kind of overhead.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @11:25AM (#19055467)
The AWS kafka product is a non-product. One would have to reproduce the same effort there as anywhere else.

60MB/s per broker

Took me about two hours worth of testing to reach that same conclusion. I like the Dropbox approach though, it is very thorough. One presumes that kafka is going to become a fundamental part of their infrastructure, in which case, this effort is a prudent investment.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by toomuchtodo on Friday February 01, 2019 @10:46AM (#19055000)
> But I'm surprised they would be wasting valuable dev time with this kind of overhead.

I'm not surprised at all. They have reached a scale where managed services are looked at with a critical eye, and are no longer the default.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by foobiekr on Friday February 01, 2019 @12:39PM (#19056319)
This. AWS is most valuable when the management overhead is a double digit percentage of your revenue. At some point the AWS margin itself is genuinely larger than DIY. That should be obvious as it has to be so unless AWS’s own costs are dramatically lower which is only possible if their value is smoke-and-mirrors.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @06:48PM (#19059902)
I don't think it's that hard to reach a scale where DIY makes sense. As was stated up above, the AWS kafka offering isn't complete. One could do something similar with a cloud formation template.

The ElasticSearch offering on the other hand is fairly complete. At my scale, it would cost me ~$15-20k/mo for ES in AWS. However, I DIY my own ES cluster (in k8s) which costs me something like $5k/mo. The delta is roughly $120-180k/yr.

An engineer costs me roughly ~$200k year with salary and benefits, so let's say the delta is .75 of an engineer. There are also risks to DIY that are hard to put an number on.

ES is definitely a PITA to run, but it's core enough to our infrastructure offering that dedicating .25-.5 of an engineer to it seems like a worthy trade-off to me. It is also a unique enough type of workload that it yields an interesting learning opportunity for my team.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by technion on Saturday February 02, 2019 @04:27AM (#19062051)
It also means you're in a position to pick up your stack and deploy it elsewhere if such a thing makes business sense in future. A lot of organisations are putting themselves in a position where that's not an option.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Sunday February 03, 2019 @12:52AM (#19068099)
If it's a given that cloud providers are aiming to get lock-in and increase margins, then it makes sense to do the necessary math to avoid that.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by toomuchtodo on Friday February 01, 2019 @08:01PM (#19060250)
I don’t disagree at all. Far too often, people wait too long to optimize away from a vendor’s offering. Their margin is your opex. Put that money back in your pocket.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by vaishaksuresh on Friday February 01, 2019 @12:30PM (#19056226)
As it stands today, AWS' managed kafka is very limited. Single region and no TLS is a non-starter for a lot of places.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by cbsmith on Friday February 01, 2019 @01:07PM (#19056652)
I'd use the Confluent managed Kafka service before I'd touch the AWS one.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @06:40PM (#19059869)
I've given it a test drive and highly approve. It is very polished.

Kinda spendy, though.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by cbsmith on Monday February 04, 2019 @05:39PM (#19081374)
Price is always negotiable. ;-)
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Zecar on Friday February 01, 2019 @10:36AM (#19054912)
> storage is their core competency

Not so sure about that. I post in every one of these threads, but they deleted our honeymoon photos on my wife's account and then deleted my account several months later without notice. I wouldn't trust them with a swap file. Move fast and break things is a terrible mantra for a file storage company.

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pwaivers on Friday February 01, 2019 @10:41AM (#19054965)
Their core product is still storage, even if you had a bad experience. You may think Dropbox is a bad product, but OP's point still stands.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by xfitm3 on Friday February 01, 2019 @10:43AM (#19054976)
Isn't their core product the interface and software you use to access their storage?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by linkmotif on Friday February 01, 2019 @10:47AM (#19055012)
Their core product is an interface, they resell cloud storage. Right?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by gitgud on Friday February 01, 2019 @10:44AM (#19054977)
Can't say that I've ever had an issue with the reliability or service at Dropbox. Not sure they're known for moving fast either, as the product has changed very little in 10 years.