Twitter open-sources a high-performance replicated log service

Re:Twitter open-sources a high-performance replicated log se... by mgodave on Tuesday May 10, 2016 @04:13PM (#11670279)

A DL stream is a single partition if you are thinking in terms of kafka. We stitch them together as a partitioned stream a la Kafka using a different system. So, to answer your question, no, there is no limit to the number of DL streams you can put together into a bigger logical stream, we have some very large ones.

[ Parent | on HN ]

Re:Twitter open-sources a high-performance replicated log se... by alexatkeplar on Tuesday May 10, 2016 @05:41PM (#11670895)

Ah! That actually explains the strange syntax I was seeing in the tutorials:

  // Create Stream `basic-stream-{3-7}`
  // dlog tool create -u ${distributedlog-uri} -r ${stream-prefix} -e ${stream-regex}
  ./distributedlog-core/bin/dlog tool create -u distributedlog://127.0.0.1:7000/messaging/distributedlog -r basic-stream- -e 3-7

You use a regex to progammatically create N streams, which would be N shards in Kinesis or N partitions in Kafka.

[ Parent | on HN ]