News for hackers. Stuff that matters.
Re:Apache Kafka Goes 1.0 by threeseed on Wednesday November 01, 2017 @03:06PM (#15604503)
Remember that both Kafka and Kinesis have "at least once" delivery and from experience it's a common scenario. You will receive multiple messages. Two options: (1) store hashes of each row and then deduplicate, (2) use a database.

I can't remember if it's an issue for change capture since in theory you have some insert timestamp column anyway.

Re:Apache Kafka Goes 1.0 by cmatta on Wednesday November 01, 2017 @03:43PM (#15604805)
Kafka has "exactly once" delivery guarantees since 0.11, I believe this will be the default in Kafka 1.0 (I work for Confluent).