News for hackers. Stuff that matters.
Re:Streaming Joins Are Hard by ryzhyk on Friday October 25, 2024 @08:09PM (#41951320)
Thanks for the kind words about DDlog :)

The reason DBSP and Differential Dataflow work so well is because they are specialized to relational computations. Relational operators have nice properties that allow evaluating them incrementally. Incremental evaluation for a general purpose language like Rust is a much, much harder problem.

FWIW, DBSP is available as a Rust crate (https://crates.io/crates/dbsp), so you can use it as an embedded incremental compute engine inside your program.

Re:Streaming Joins Are Hard by incr_me on Saturday October 26, 2024 @02:56AM (#41953061)
Indeed. I've experimented a bit with abusing DD/DBSP for my purposes by modeling various kinds of data structures in terms of Z-sets, but these efforts have not yielded very impressive results. :)

For how elegant DBSP is I still found the paper a tough nut to crack, and it really is one of the more accessible theoretical contributions in the space, at least from this grubby programmer's perspective... I hope to devote some time to study and play around more, but in the meantime I'm rooting for you!

Re:Streaming Joins Are Hard by ryzhyk on Saturday October 26, 2024 @12:57PM (#41955999)
Thanks again!

You may want to check out this tutorial for a hands-on introduction to DBSP: https://docs.rs/dbsp/0.28.0/dbsp/tutorial/index.html

Re:Streaming Joins Are Hard by ryzhyk on Saturday October 26, 2024 @01:01PM (#41956031)
Also there's now a DBSP implementation in pure Python! https://github.com/brurucy/pydbsp