Issues we've encountered while building a Kafka based data processing pipeline

Re:Issues we've encountered while building a Kafka based dat... by BiteCode_dev on Monday October 18, 2021 @11:32AM (#28906491)

Well, if you want events, you have lighter alternative. Redis pub/sub, crossbar... Even rabbitMQ is lighter.

If you need queuing, you have libs like celery.

You don't need to go full kafka

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by sumtechguy on Tuesday October 19, 2021 @09:27AM (#28917843)

I do not disagree. Kafka is kind of a monster to fully configure correctly. Once they get rid of zookeeper it may be nicer to spin up and 'set it and forget it' sort of thing like the others.

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by emerongi on Monday October 18, 2021 @02:22PM (#28909190)

Redis - it's probably in your stack already and fills all the use-cases.

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by anotherhue on Monday October 18, 2021 @07:18PM (#28913063)

Agreed. Redis pub/sub is underrated.

[ Parent | on HN ]