Re:Why isn't differential dataflow more popular?
by scott_s on Monday January 25, 2021 @11:18AM
(#25904912)
Myself and a few others have done a lot of research on performing sliding window aggregations updates without recomputing everything. Our code is on github, and the README has links to the papers: https://github.com/IBM/sliding-window-aggregators
o Generally, we do not want to re-compute aggregates - aggregates should be also updated, particularly, if n is very large
o In real applications, rolling aggregation is performed using partitioning on some objects. For example, we append new events from many different devices to one table and want to compute rolling aggregates for each individual device. Hence, this (i-n, i+n) will not work anymore.
o Rolling aggregation using absolute time windows will also work differently. Although, if records are ordered (like in stream processing) and there are no partitions, then it is easy.