The new local data stack: Integrating Dagster, dbt and DuckDB

Re:The new local data stack: Integrating Dagster, dbt and Du... by Jgrubb on Thursday December 14, 2023 @09:21AM (#38641596)

Hey Georg, thanks for posting. I've been working on building a thing with almost the same stack lately and the Dagster integration is my next large-ish step. In the last couple weeks I've kicked around Duck and Clickhouse for the backend. It's been a lot of fun. Ultimately I'd like something that can be run locally or easily installed and run on a PaaS.

The only trick with local is that data sets of any appreciable size take ages to pull down, at least here in the US with our terrible internet (in the average Italian mountain village this would probably work great).

[ Parent | on HN ]

Re:The new local data stack: Integrating Dagster, dbt and Du... by geoHeil on Thursday December 14, 2023 @12:33PM (#38644203)

Well local can mean your laptop. But could mean your local server. However, it can also mean a VM on your cloud provider of choice colocated to the object store.

In such a case the network transfer (for many cases) can be almost irrelevant - at least for up to medium-ish sized datasets.

[ Parent | on HN ]

Re:The new local data stack: Integrating Dagster, dbt and Du... by Jgrubb on Friday December 15, 2023 @01:54PM (#38657277)

Funny thing, I'm trying to get going with Dagster today like I said, and figuring out how to debug is a bit .. annoying. The most helpful post I've come across so far just happens to be https://georgheiler.com/2022/02/02/interactive-dagster-debug...

So thanks!

[ Parent | on HN ]