News for hackers. Stuff that matters.
Re:Finding Kafka’s throughput limit in Dropbox infrastructure by andmarios on Friday February 01, 2019 @01:36PM (#19056998)
I think it's a good number for a sustained load within a large cluster.

50MB/sec multiplies due to replication and consumers. Also in large clusters, the bottleneck comes often from the network switches rather than the rest of the stack.

Also when it comes to sustained load, many people don't realize that 50MB/sec × 10 brokers × 24 hours × 7 days × 3600 seconds leads to 300TB per week (without counting the replication). :)

Re:Finding Kafka’s throughput limit in Dropbox infrastructure by zamadatix on Saturday February 02, 2019 @06:57PM (#19066731)
32 port 100G switches are dirt cheap. If the network is your bottleneck you likely overspent on the wrong hardware.