News for hackers. Stuff that matters.
Re:Streaming Joins Are Hard by crazygringo on Friday October 25, 2024 @05:49PM (#41950147)
I definitely understand the value of streaming. Your gauges example is great.

What I don't understand is streaming joins. None of your gauge values need to join to anything.

And if they did -- if something needed to join ID values to display names, presumably those would sit in a database, not a different stream?

Re:Streaming Joins Are Hard by majormajor on Friday October 25, 2024 @09:24PM (#41951726)
> And if they did -- if something needed to join ID values to display names, presumably those would sit in a database, not a different stream?

At a high level the push-instead-of-pull benefit here is "you don't have to query the ID values to get the display names every time" which will reduce your latency. (You can cache but then you might get into invalidation issues and start thinking "why not just send the updates directly to my cache instead")

There's also a less cacheable version where both sides are updating more frequently and you have logic like "if X=1 and Y=2 do Z."

For small enough batches streaming and micro-batching do often end up very similar.