Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by pandemic_region on Friday February 01, 2019 @10:48AM
(#19055022)
Line is doing 150 billion msgs/day on their Kafka infra, about 3million per second.
https://www.slideshare.net/linecorp/building-a-companywide-d...
[ Parent
| on HN ]
Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by buro9 on Friday February 01, 2019 @11:24AM
(#19055461)
At Cloudflare a year ago we said we were doing 6m per second ingest rate on a cluster of 106 brokers with x3 replication factor, 106 partitions: https://blog.cloudflare.com/http-analytics-for-6m-requests-p...
The more interesting blog post IMHO is this one on Kafka compression https://blog.cloudflare.com/squeezing-the-firehose/
Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by Rebelgecko on Friday February 01, 2019 @12:23PM
(#19056136)
The compression article was really useful for me when I was trying to squeeze a bit more performance out of our brokers at work. Any plans to revisit the article now that Kafka has zstd support?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by nosequel on Friday February 01, 2019 @12:58PM
(#19056549)
They talk about zstd in the post and they were the ones who added support to Sarama.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure
by halbritt on Friday February 01, 2019 @06:39PM
(#19059861)
I'm using zstd with my kafka implementation and it works great. v2.1 is required. We're using librdkafka, which works, but I believe we had to compile it in to get that and idempotent producer.