Re:An opinionated map of incremental and streaming systems
by jamii on Tuesday May 04, 2021 @07:11PM
(#27044000)
This is why the high temporal locality part of the map is all EC - when everything is windowed you can just wait for the window to close.
On the low temporal locality side, the only consistent system I've seen so far is differential dataflow (and materialize) which does internally track which results are consistent and gives you the option to wait for consistent results in the output.
Re:An opinionated map of incremental and streaming systems
by staticassertion on Tuesday May 04, 2021 @03:27PM
(#27041824)
Exactly. People act like EC is impossible to make viable, and ignore the fact that transactional logic can impose 10-100x performance loss. Putting an EC system in front of a transactional system can be a massive performance win.
This isn't quite what the Helios paper talks about, but there's lots of things like async indexing in there that are kinda similar in nature
If you're doing any kind of Important Reporting on eventually consistent data, you'd better make sure that you either know you're only including finalised data, or that there's a big fat warning with error bars.