Re:Hydro: Distributed Programming Framework for Rust
by tomnicholas1 on Friday January 31, 2025 @07:07AM
(#42886837)
I was also going to say this looks similar to one layer of dask - dask takes arbitrary python code and uses cloudpickle to serialise it in order to propagate dependencies to workers, this seems to be an equivalent layer for rust.
[ Parent
| on HN ]
Re:Hydro: Distributed Programming Framework for Rust
by FridgeSeal on Friday January 31, 2025 @05:20PM
(#42892950)
This looks to be a degree more sophisticated than that.
Authors in the comments here mention that the flo compiler (?) will accept-and-rewrite Rust code to make it more amenable to distribution. It also appears to be building and optimising the data-flow rather than just distributing the work. There’s also comparisons to timely, which I believe does some kind of incremental compute.