Publishing with Apache Kafka at The New York Times

Re:Publishing with Apache Kafka at The New York Times by pram on Wednesday September 06, 2017 @06:17PM (#15187583)

I'd say it isn't appropriate for permanent data storage because the individual brokers don't scale well with the amount of logs present. If you have hundreds of partitions, and millions of logs, then any operation dealing with the indexes (like an unclean startup) will take an extremely long amount of time. So your individual brokers are now down for an hour if they don't shut down cleanly. Which happens often (oom, weird zk errors, etc)

It scales linearly.

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by LgWoodenBadger on Wednesday September 06, 2017 @11:05PM (#15188889)

Most of the use-cases you describe wouldn't be resolved by re-reading the entirety of the topics from the beginning of time.

They'd resume from their last committed offset and continue where they left off.

If you do have to catastrophically recover from the beginning of time, then sure you'd have a rough time. But that's true for any system that would have to do that. It's not Kafka-specific.

Now if your consumer was entirely in-memory and made no use of persistent storage itself and it had to recover from the beginning of time, then I'd say that type of problem is your own architectural failure and I have yet to come across a tool, pattern, framework, or architecture that bullet-proofs your foot.

[ Parent | on HN ]

Re:Publishing with Apache Kafka at The New York Times by pram on Thursday September 07, 2017 @02:02AM (#15189453)

Reading the topic from the beginning is not what I'm talking about.

[ Parent | on HN ]