News for hackers. Stuff that matters.
Re:Hydro: Distributed Programming Framework for Rust by stefanka on Friday January 31, 2025 @04:23AM (#42886032)
How does this compare to timely [0] in terms of data flow? Can you represent control flow like loops in the IR?

[0] https://github.com/TimelyDataflow/timely-dataflow

Re:Hydro: Distributed Programming Framework for Rust by leicmi on Friday January 31, 2025 @07:59AM (#42887214)
Their latest paper [0] refers to Naiad(timely dataflow) a few times, e.g.: "Inspired by ingress/egress nodes in Naiad [34], nested streams can be processed by nested dataflow graphs, which iteratively process chunks of data sourced from a larger stream with support for carrying state across iterations."

[0] https://hydro.run/papers/flo.pdf

Re:Hydro: Distributed Programming Framework for Rust by tel on Friday January 31, 2025 @11:16AM (#42888963)
Reading a bit about it from the Flo paper

- Describe a dataflow graph just like Timely - Comes from a more "semantic dataflow" kind of heritage (frp, composition, flow-of-flows, algebraic operators, proof-oriented) as opposed to the more operationally minded background of Timely - Has a (very) different notion of "progress" than Timely, focused instead of ensuring the compositions are generative in light of potentially unbounded streaming inputs - In fact, Flo doesn't really have any notion of "timeliness", no timestamping at all - Supports nested looping like Timely, though via a very different mechanism. The basic algebra is extremely non-cyclic, but the nested streams/graphs formalism allows for iteration.

The paper also makes a direct comparison with DBSP, which as I understand it, is also part of the Timely/Naiad heritage. Similar to Timely, the authors suggest that Flo could be a unifying semantic framework for several other similar systems (Flink, LVars, DBSP).

So I'd say that the authors of Flo are aware of Naiad/Timely and took inspiration of nested iterative graphs, but little else.

Re:Hydro: Distributed Programming Framework for Rust by v3xro on Friday January 31, 2025 @11:47AM (#42889244)
Thanks for the summary, really looks like something that is worth digging into!
Re:Hydro: Distributed Programming Framework for Rust by shadaj on Friday January 31, 2025 @01:02PM (#42889931)
Flo lead-author here! This is spot on :) Flo aims to be a bit less opinionated than Timely in how the runtime should behave, so in particular we don't support the type of "time-traveling" computation that Timely needs when you have iterative computations on datasets with retractions.

This is also one of the core differences of Timely compared to DBSP, which uses a flat representation (z-sets) to store retractions rather than using versioned elements. This allows retractions to be propagated as just negative item counts which fits into the Flo model (and therefore Hydro).