Facebook open-sources LogDevice, a distributed storage for sequential data

Re:Facebook open-sources LogDevice, a distributed storage fo... by beepbeepbeep1 on Thursday September 13, 2018 @06:37PM (#17983315)

It does not, I've lost alot of time profiling Kafka perf issues against clusters on the exact same hardware with exact same traffic but with a 3000% throughput difference. The root cause was one cluster had a lot of empty test topics

Try benchmarking Kafka from 0 partitions to a few thousand partitions in 100 partition increments. The benchmark only needs to write to a single topic, using their provided producer perf tool while all other topics are inactive with zero data.

As the partitions increase there is a very noticeable drop in throughout that looks to be linear.

Kafka does not handle a large number of partitions well currently, large even being low thousands. It's easy to hit with just a few hundred topics.

Reading between the lines ehen Linkdin and Netflix advertise several clusters, i am predicting/guessing they shard the data.

[ Parent | on HN ]

Re:Facebook open-sources LogDevice, a distributed storage fo... by otterley on Friday September 14, 2018 @12:10PM (#17988378)

I didn't think we were speaking of millions of topics or partitions here; only millions of logs. You can certainly have logs numbering in the millions using a single topic. Mux/demux would have to happen at the producer/consumer side, of course.

[ Parent | on HN ]