News for hackers. Stuff that matters.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by pandemic_region on Friday February 01, 2019 @10:48AM (#19055022)
Line is doing 150 billion msgs/day on their Kafka infra, about 3million per second. https://www.slideshare.net/linecorp/building-a-companywide-d...
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by buro9 on Friday February 01, 2019 @11:24AM (#19055461)
At Cloudflare a year ago we said we were doing 6m per second ingest rate on a cluster of 106 brokers with x3 replication factor, 106 partitions: https://blog.cloudflare.com/http-analytics-for-6m-requests-p...

The more interesting blog post IMHO is this one on Kafka compression https://blog.cloudflare.com/squeezing-the-firehose/

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by Rebelgecko on Friday February 01, 2019 @12:23PM (#19056136)
The compression article was really useful for me when I was trying to squeeze a bit more performance out of our brokers at work. Any plans to revisit the article now that Kafka has zstd support?
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by nosequel on Friday February 01, 2019 @12:58PM (#19056549)
They talk about zstd in the post and they were the ones who added support to Sarama.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by halbritt on Friday February 01, 2019 @06:39PM (#19059861)
I'm using zstd with my kafka implementation and it works great. v2.1 is required. We're using librdkafka, which works, but I believe we had to compile it in to get that and idempotent producer.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by tudelo on Friday February 01, 2019 @11:21AM (#19055421)
That seems... like a lot. Thanks for the link.