Issues we've encountered while building a Kafka based data processing pipeline

Re:Issues we've encountered while building a Kafka based dat... by bsaul on Monday October 18, 2021 @06:54AM (#28904126)

using a sql db for push/pop semantic feels like using a hammer to squash a bug.. How would you model queues & partitions with ordering guarantees with pg ?

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by throwaway81523 on Monday October 18, 2021 @07:53AM (#28904480)

With transactions, and stored procedures if that helps ;). Redis also seems well suited to the use cases I've seen for Kafka. Kafka must have capabilities beyond those use cases, and I've sometimes wondered what they are.

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by anotherhue on Monday October 18, 2021 @06:57AM (#28904140)

sql has many conveniences for doing so, it wouldn't be much work.

> using a hammer to squash a bug..

Agreed - but Kafka is a much much bigger hammer. SES/Az Queues are also good choices.

[ Parent | on HN ]