News for hackers. Stuff that matters.

A comment on Publishing with Apache Kafka at The New York Times. Parent comment

Re:Publishing with Apache Kafka at The New York Times by theossuary on Wednesday September 06, 2017 @03:39PM (#15186510)
See, that's where I'm confused. I'm no Kafka expert, but they say they use a "single-partition topic" which I believe means the only way they can replicate the data is by replicating the entire log, they can't shard because it's a single partition. The reasoning behind this is because Kafka doesn't support ordering between partitions.

Also I've never thought of Kafka as a persistent data storage solution, it's interesting Confluent is supporting Kafka being used in this way.

[ Parent | on HN ]
Re:Publishing with Apache Kafka at The New York Times by tlberglund on Wednesday September 06, 2017 @09:57PM (#15188614)
You are not even a little bit confused! I think you have it perfectly right. And you are not alone in not thinking of Kafka as persistent storage, but when you get down to it there is no reason not to, and people are indeed using it in just that way. And yes, Confluent does give its +1 to this practice. :)
[ Parent | on HN ]
Re:Publishing with Apache Kafka at The New York Times by poooogles on Wednesday September 06, 2017 @04:49PM (#15187016)
>it's interesting Confluent is supporting Kafka being used in this way.

If it earns them money, I think they'll pretty much support anything. Jay has rubber stamped it on SO [1], but he's got a bit of a vested interest on selling Kafka.

1. https://stackoverflow.com/questions/17708489/using-kafka-as-...

[ Parent | on HN ]