News for hackers. Stuff that matters.
Re:ArkFlow: High-performance Rust stream processing engine by habobobo on Tuesday April 29, 2025 @11:32AM (#43834033)
Looks interesting, how does this compare to arroyo and vector.dev?
Re:ArkFlow: High-performance Rust stream processing engine by tormeh on Tuesday April 29, 2025 @11:54AM (#43834353)
Also curious about any comparison to Fluvio.
Re:ArkFlow: High-performance Rust stream processing engine by necubi on Tuesday April 29, 2025 @01:03PM (#43835238)
(I'm the creator of Arroyo)

I haven't dug deep into this project, so take this with a grain of salt.

ArkFlow is a "stateless" stream processor, like vector or benthos (now Redpanda Connect). These are great for routing data around your infrastructure while doing simple, stateless transformations on them. They tend to be easy to run and scale, and are programmed by manually constructing the graph of operations.

Arroyo (like Flink or Rising Wave) is a "stateful" stream processor, which means it supports operations like windowed aggregations, joins, and incremental SQL view maintenance. Arroyo is programmed declaratively via SQL, which is automatically planned into a dataflow (graph) representation. The tradeoff is that state is hard to manage, and these systems are much harder to operate and scale (although we've done a lot of work with Arroyo to mitigate this!).

I wrote about the difference at length here: https://www.arroyo.dev/blog/stateful-stream-processing