News for hackers. Stuff that matters.
Re:Feldera Incremental Compute Engine by jonstewart on Sunday September 29, 2024 @09:42AM (#41687358)
How does it compare to Materialize/differential dataflow?
Re:Feldera Incremental Compute Engine by lsuresh on Sunday September 29, 2024 @10:33AM (#41687690)
(Feldera's CEO here)

We are based on DBSP (https://www.vldb.org/pvldb/vol16/p1601-budiu.pdf) which is an evolution of DD. DBSP gives us an algorithm to take arbitrarily complex queries and generate an incremental version of it.

As a consequence, we evaluate everything incrementally. For example, we are the only engine that can perform rolling aggregates incrementally. In general, with DBSP, we can incrementalize "the full relational algebra, queries over sets and multisets, arbitrarily nested relations, aggregation, flatmap (unnest), monotonic and non-monotonic recursion, streaming aggregation, and arbitrary compositions of all of these". DBSP is a much simpler and cleaner foundation.

As a product, both our enterprise and open-source (MIT licensed) offerings let you run it anywhere you want including your laptop.

Positioning wise, we are a query engine with a unified way to compute over both bounded and unbounded data sources with perfect consistency, with an integrated storage layer. Materialize is going for building an operational warehouse.

Re:Feldera Incremental Compute Engine by ZiliangXK on Sunday September 29, 2024 @02:48PM (#41689475)
There is always a tipping point for quite a few use cases where incremental evaluation degrades perf compared with full batch / historical query ?
Re:Feldera Incremental Compute Engine by jonstewart on Sunday September 29, 2024 @03:09PM (#41689606)
Thank you! I will look into it further.