News for hackers. Stuff that matters.
Re:Feldera Incremental Compute Engine by shuaiboi on Sunday September 29, 2024 @03:29PM (#41689734)
Just a guess... wouldl like to hear the answer as well.

they probably have a monotonicity detector somewhere, which can decide whether to keep all the values or discard them. If they keep them, they probably use something like a segment tree to index.

Re:Feldera Incremental Compute Engine by lsuresh on Sunday September 29, 2024 @03:45PM (#41689856)
Yes, we do a lot of work with monotonicity detection. It's central to we perform automatic garbage collection based on lateness.
Re:Feldera Incremental Compute Engine by ryzhyk on Sunday September 29, 2024 @06:26PM (#41691342)
That's right, we perform static dataflow analysis to determine what data can get discarded. GC itself is done lazily as part of LSM tree maintenance. For MAX specifically, we don't have this optimization yet. In the general case, incrementally maintaining the MAX aggregate in the presence of insertions and deletions requires tracking the entire contents of the group, which is what we do. If the collection can be proved to be append-only, then it's sufficient to store only the current max element. This optimization is yet coming to Feldera.