Show HN: Data load tool(dlt)-Python library to automate the creation of datasets

Re:Show HN: Data load tool(dlt)-Python library to automate t... by adrianbr on Wednesday October 25, 2023 @05:14AM (#38010841)

Since dlt generates a schema, and tracks evolution etc, contains lineage, and follows data vault standard it can easily provide metdata or lineage info to the other tools.

At the same time, dlt is a pipeline building tool first - so if people want to read metadata from somewhere and store it elsewhere, they can.

If you mean to take metadata like we integrate with arrow - that remains to be seen if the community might want this or find it useful, we will not develop plugins for collecting cobwebs, but if there are interested users we will add it to our backlog.

[ Parent | on HN ]

Re:Show HN: Data load tool(dlt)-Python library to automate t... by pitah1 on Wednesday October 25, 2023 @05:36AM (#38010983)

Thanks for the response. I also noticed there was a mention of data contracts or Pydantic to keep your data clean. Would it make sense to embed that as part of a DLT pipeline or is the recommendation to include it as part of the transformation step?

[ Parent | on HN ]

Re:Show HN: Data load tool(dlt)-Python library to automate t... by MatthausK on Wednesday October 25, 2023 @06:14AM (#38011214)

You can use pydantic models to define schemas, validate data (we also load instances of the models natively): https://dlthub.com/docs/general-usage/resource#define-a-sche...

We have a PR (https://github.com/dlt-hub/dlt/pull/594) that is about to merge that makes the above highly configurable, between evolution and hard stopping: - you will be able to totally freeze schema and reject bad rows - or accept the data for existing columns but not new columns - or accept some fields based on rules'

[ Parent | on HN ]

Re:Show HN: Data load tool(dlt)-Python library to automate t... by adrianbr on Wednesday October 25, 2023 @05:15AM (#38010848)

you can request a source or a feature by opening an issue on sources/dlt repo https://github.com/dlt-hub

[ Parent | on HN ]