Re:What If We Could Rebuild Kafka from Scratch?
by dpflan on Friday April 25, 2025 @04:43PM
(#43798318)
Thanks. With these of examples of messages appearance in time and in physical location in Kafka, how have you adapted your consumers? Which scenario / architectural decision (one of the examples?) have you moved forward with and creating support to yield your desired causality handling?
[ Parent
| on HN ]
Re:What If We Could Rebuild Kafka from Scratch?
by galeaspablo on Friday April 25, 2025 @08:19PM
(#43799758)
Option 1, but after so many years banging our heads against the wall reasoning about this, we hoped someone would eventually give us a queue that supports arbitrary causal dependency graphs.
We thought about building it ourselves, because we know the data structures, high level algorithms, and disk optimizations required. BUT we pivoted our company, so we've postponed this for the foreseeable future. After all, theory is relatively easy, but a true production grade implementation takes years.