What If We Could Rebuild Kafka from Scratch?

Re:What If We Could Rebuild Kafka from Scratch? by taurath on Saturday April 26, 2025 @06:42PM (#43807880)

Wanted to say thanks so much for writing this all out - I've always thought of ordering as being sort of inherently against the point of parallel streams, so its interesting to hear about the state of the art and the benefits that are trying to be gleaned! I'm not thinking in stream processors terribly often so I wasn't aware of how dependencies are mapped.

If you don't mind another followup (and your patience with my ignorance hasn't run out :P), wouldn't the efficient concurrent consumption imply knowing the dependency graph before the events are processed? IE, is it possible in any instance to get to O(w+h) in a stream?

[ Parent | on HN ]

Re:What If We Could Rebuild Kafka from Scratch? by galeaspablo on Sunday April 27, 2025 @01:40PM (#43813719)

No problem. :)

Yes, order needs to be known.

So no, it’s not possible to do O(w+h) with streams partitioned by key. Unless, of course you use a supplementary index, but then you might as well not use the streams storage at all and store the records in the same storage as the index.

It’s worth noting that Pulsar does something like this (supplementary way to keep track of acknowledged messages), but their implementation has O(n^2) edge cases.

[ Parent | on HN ]