Issues we've encountered while building a Kafka based data processing pipeline

Re:Issues we've encountered while building a Kafka based dat... by Spivak on Tuesday October 19, 2021 @12:00AM (#28914723)

You don’t want stream replication for scale, you want it for availability and durability. The number of times replication has saved my ass is too damn high.

And you want Kafka (or something like it) the moment you need two processes handling updates which again you probably want for availability reasons so a crash doesn’t stop the stream.

You also don’t catch deletes or updates with this setup. But there’s a million db to Kafka connectors that read the binlog or pretend to be a replica to handle this.

[ Parent | on HN ]

Re:Issues we've encountered while building a Kafka based dat... by BiteCode_dev on Tuesday October 19, 2021 @04:28AM (#28916005)

Sure, if you need high availability + replication cross db, go for Kafka, but that's the thread point: it's not a common use case.

Most people don't need high availability, and most replication use the same db which comes with tooling for that.

[ Parent | on HN ]