News for hackers. Stuff that matters.
Re:Feldera Incremental Compute Engine by shuaiboi on Sunday September 29, 2024 @02:51PM (#41689492)
would something like dbsp support spreadsheet style computations? Most of the financial world is stuck behind spreadsheets and the entire process of productioinizing spreadsheets is broken:

* Engineers don't have time to understand the spreadsheet logic and translate everything into an incremental version for production.

* Analysts don't understand the challenges with stream processing.

* SQL is still too awkward of a language for finance.

* Excel is a batch environment, which makes it hard to codify it as a streaming calculation.

If I understand correctly, your paper implies as long as there is a way to describe spreadsheets as a Zset, some incremental version of the program can be derived? Spreadsheets are pretty close to a relational table, but it would be a ZSet algebra on cells, not rows, similar to functional reactive programming. So dbsp on cells would be incremental UDFs, not just UDAFs?

thoughts??

Re:Feldera Incremental Compute Engine by lsuresh on Sunday September 29, 2024 @03:54PM (#41689923)
Great question. DBSP should work here -- spreadsheets are by definition incremental (and there's even recursive queries there with cells depending on each other).

Note that we use Z-Sets to bridge SQL/tables with DBSP, but Z-Sets aren't general enough for spreadsheets.