Re:Kafka Is Costing You Years of Engineering Time
by galeaspablo on Friday October 04, 2024 @10:22PM
(#41747262)
Hey Denis, I haven’t run into you before. But hi, this is Luis, Ambar’s founder. Nice to meet a fellow data streamer.
When I started writing Ambar I thought streaming from a database was a solved problem. But in operational use cases where ordering and delivery guarantees are assumptions developers need, it isn’t a solved problem. The first version of Ambar was just Debezium under the hood, but guess what, it failed and failed hard. Like you described Kafka. Hence we built Ambar :)
FYI we’ve considered using Redpanda under the hood instead of Kafka, but didn’t dare make the jump yet.
Re:Kafka Is Costing You Years of Engineering Time
by deniscoady on Friday October 04, 2024 @10:40PM
(#41747336)
Ah okay, so is Ambar more of a way to finally replace Debezium then?
Re:Kafka Is Costing You Years of Engineering Time
by galeaspablo on Saturday October 05, 2024 @07:10AM
(#41749071)
Yes, for operational use cases. Eg event driven microservices communication. Keeping in mind we replace the sink as well, which allows us to do cool things such as https://ambar.cloud/blog/optimal-consumption-with-adaptive-l...
For analytics (eg copy your PG database to Snowflake), Debezium is still relevant.
I've worked with Apache Kafka at massive (50+ Gbps) scales. It's a proper nightmare. When it breaks – it breaks fast and violently.
But the problem is that Apache Kafka (and more modern Kafka-compatible alternatives like Redpanda < obligatory mention) solve a need for a durable streaming log that other systems cannot offer. The access patterns, requirements, use cases, ecosystem, etc, are different from those of traditional databases and require a proper streaming solution.
Streaming from a traditional database is kinda a solved problem. Why not just use a managed Kafka provider with a change data capture (CDC) capability if you don't want to deal with Kafka yourself? At least then you get to use all of the tools in the vibrant Kafka ecosystem.