News for hackers. Stuff that matters.

A comment on Show HN: Data load tool(dlt)-Python library to automate the creation of datasets.

Re:Show HN: Data load tool(dlt)-Python library to automate t... by Terretta on Wednesday October 25, 2023 @07:37AM (#38011706)
> dlt is a minimalistic Python library, requires no backend and works whenever Python works. You can finetune it to ... run with ... Dagster.

Relating to dagster in particular, this is in your docs:

dlt incorporates the concept of implicit extraction DAGs to handle the dependencies between data sources and their transformations automatically. A DAG represents a directed graph without cycles, where each node represents a data source or transformation step.

When using dlt, the tool automatically generates an extraction DAG based on the dependencies identified between the data sources and their transformations. This extraction DAG determines the optimal order for extracting the resources to ensure data consistency and integrity.

How do you think about tying and running this and dagster together?

[ Parent | on HN ]
Re:Show HN: Data load tool(dlt)-Python library to automate t... by adrianbr on Wednesday October 25, 2023 @07:54AM (#38011806)
There are multiple ways to run together - we will show a few in a demo coming out soon.

We also consider a tighter integration like with Airflow described here as a possible next step https://dlthub.com/docs/walkthroughs/deploy-a-pipeline/deplo...

We will investigate the interest incrementally as to not build any plugins that don't end up used.

[ Parent | on HN ]
Re:Show HN: Data load tool(dlt)-Python library to automate t... by nerdponx on Wednesday October 25, 2023 @01:49PM (#38015791)
For an example of prior art, you should look into Astronomer's Cosmos Library to see how they integrate Dbt into Airflow.
[ Parent | on HN ]
Re:Show HN: Data load tool(dlt)-Python library to automate t... by adrianbr on Wednesday October 25, 2023 @02:28PM (#38016388)
Thank you! That's the example we looked at for our dlt-airflow integration :) the dlt dag becomes an airflow dag.
[ Parent | on HN ]