Show HN: Arroyo – Write SQL on streaming data

Re:Show HN: Arroyo – Write SQL on streaming data by sorenbs on Wednesday June 07, 2023 @07:28AM (#36225507)

This is a really exciting project! I recently learned about https://github.com/vmware/database-stream-processor which builds on a new theoretical foundation and claims to be 9x faster than Flink. It is also written in Rust, and there is a compiler from SQL to Rust executables. Can you comment on the differences?

[ Parent | on HN ]

Re:Show HN: Arroyo – Write SQL on streaming data by sorenbs on Wednesday June 07, 2023 @07:50AM (#36225649)

As an aside - we are evaluating different streaming engines to power data projections for Prisma.io. Excited to see support for Debezium coming in v0.4.0.

Would be interesting to somehow make Arroyo run the Nexmark benchmark so we can clearly compare to Flink and DBSP: https://liveandletlearn.net/post/vmware-take-3-experience-wi...

[ Parent | on HN ]

Re:Show HN: Arroyo – Write SQL on streaming data by jnewhouse on Wednesday June 07, 2023 @08:39AM (#36226030)

Hi there! We actually already have a built-in Nexmark source. It's pretty useful for developing new capabilities, and available as a source out of the box.

Just read through the DBSP docs and it looks like it is working in a similar space. The biggest differences in my mind are around distribution and reliability. Arroyo works across a cluster of machines and has built in fault tolerance, while for DBSP that's still just planned for the future.

(I'm the corrector of Arroyo, for context)

[ Parent | on HN ]

Re:Show HN: Arroyo – Write SQL on streaming data by sorenbs on Wednesday June 07, 2023 @10:52AM (#36227521)

Thank you!

I'll try to get something set up to compare performance of the two on the same machine.

[ Parent | on HN ]

Re:Show HN: Arroyo – Write SQL on streaming data by necubi on Wednesday June 07, 2023 @12:34PM (#36229235)

That'd be great! We have versions of most of the nexmark queries and some internal benchmark vs Flink, we'd love to help if you're interested in benchmarking against more systems. Reach out at [email protected].

[ Parent | on HN ]

Re:Show HN: Arroyo – Write SQL on streaming data by ryzhyk on Wednesday June 07, 2023 @01:10PM (#36229789)

I'm a developer of DBSP. Our repo now lives here: https://github.com/feldera/dbsp/. And here is some more benchmarking data vs Flink and Beam: https://github.com/feldera/dbsp/tree/main/benchmark.

See feldera.com for more info.

[ Parent | on HN ]

Re:Show HN: Arroyo – Write SQL on streaming data by sorenbs on Tuesday June 13, 2023 @07:59AM (#36308958)

Thank you!

Really happy to see all the activity at the new repo. For a while there I thought the project was dying...

[ Parent | on HN ]