amath

Congrats on the release. Can dstack work with kubernetes? Like say another team owns the infrastructure and I’m required to use k8s, is it interoperable at all?

This seems like a good model for sustaining open source, but raises some questions.

Does anybody know how the DuckDB foundation works? The sponsors are MotherDuck, Voltron, and Posit, which are heavily venture-funded. Do DuckDB Lab employees work on sponsored projects for the foundation?

I am also curious if anyone can shed light on what kind of contract work DuckDB does to align its work with the open source project. This has always seemed like the holy grail, but it is difficult to do in practice.

This is just that authors opinion. It seems clear that there is a spectrum of what users want from basic retrieval to content generation.

Sounds like you are writing a pipeline to enrich a stream of data with historical data from Snowflake. This is a fairly common pattern. If the data in snowflake is not changing often, you would want to cache that somewhere for use with the stream processor to avoid that query overhead and speed things up.

Most of those listed can meet your first 2 requirements. Looking further down the list, your requirement of SQL and a DAG type of representation will limit the list to only a few. I don't know if many of those listed provide both of those capabilities.

If you relax the SQL constraint, more of them are applicable like Bytewax and Kafka-streams.

Developer Voices podcast is one of my favorites.

More interested in the opinions of those on this site and I thought the article would prompt some interesting discussion.

Nice work on the performance boost :).

How does it compare with things like: 1. https://github.com/bytewax/bytewax 2. https://github.com/pathwaycom/pathway

I recently read this article (https://materializedview.io/p/from-samza-to-flink-a-decade-o...) about Flink and it commented on Flink grew to fit all of these different use cases (applications, analytics and ETL) with disjoint requirements that Confluent built kafka-streams, ksql and connector for. What of those would you say Arroyo is better suited for?

Maybe this is a stupid question, but how would airbnb be looked at using this as precedent. If I sign up for airbnb and use their automatic price optimization engine would this fall under the same algorithmic collusion?

Great article! The breakdown of ETL, Analytical and Production workloads is so critical to understanding the pieces of the ecosystem and the history from your experience makes it much easier to understand.

I'm always curious about who writes all the comments on Hacker News. I am more of a lurker than a commenter.

We built this live Grafana Dashboard that uses Bytewax and Proton to pull data from the Hacker News API, process it as a stream, and then present it in a live updating dashboard.

I can't tell if you think that's a bad thing. I see it often, but yah this is a whole tutorial in the README.

Ah, this had me laughing.

Maybe this is a contradictory thought, but this is probably a good thing. I think you could argue that most drug mules are doing it because they don't, or don't think, they have another option to change their current life. If this reduces the number of humans used to smuggle drugs, it is probably a good thing.

These types of stories are what drove me to software in the first place. The ideal of a group of people working towards the same goal, for sometimes different reasons, but not for the sole purpose of monetary reward.

Good luck with the project!

I also echo some of the others, it would be great to see a sort of comparison with the alternatives so that we can better understand how the project fits in.

I worked at GitHub and this is exactly what I always wanted as someone working with data and ML artifacts. We even prototyped something with GitLFS once that looked like a very hacky and less well-thought-out version of this. Great work.

I am often confused on how things like imperative stream processing frameworks are supposed to work with the streaming database, if at all. Like what are you supposed to interact with the streaming database with? Is it only for streaming ETL? Do you need another application layer that can natively integrate, orchestrate and function. Maybe using a stream processing framework with a streaming database could facilitate a full application stack.

Thanks for sharing! It does seem like a lot of trouble for little if any benefit.