Re:Facebook open-sources LogDevice, a distributed storage fo...
by manigandham on Thursday September 13, 2018 @06:36PM
(#17983306)
Kafka is also very embedded everywhere now, with a big first-mover advantage. Pulsar already does everything Kafka does but also supports custom functions, per-message acknowledgements, and native cross-region replication.
Unfortunately it's hard to change something that already works. Most users don't hit the performance limits of their tools so they'll just continue Kafka if it's already running.
In Kafka bulk reading is very cheap, the broker basically just calls sendfile() to send a file segment with compressed message chunks. On the other hand only the leader of a partition can serve requests, so you are often limited by bandwidth. It looks like LogDevice has to do a bit more work server side, but may be able to read from all servers with a replica.
Kafka stores more metadata in the record wrapper, like client and server timestamps and partition key.
There are client libraries for C++ and Python.
Operationally they look similar - both require a Zookeeper cluster, and both require assigning permanent ids to nodes.
It would be interesting to see some benchmarks comparing LogDevice with Kafka and Pulsar. That said, I suspect from the lack of buzz around Pulsar that Kafka isn't a performance bottleneck for most people using it.