News for hackers. Stuff that matters.

A comment on Show HN: Data load tool(dlt)-Python library to automate the creation of datasets. Parent comment

Re:Show HN: Data load tool(dlt)-Python library to automate t... by pitah1 on Wednesday October 25, 2023 @05:36AM (#38010983)
Thanks for the response. I also noticed there was a mention of data contracts or Pydantic to keep your data clean. Would it make sense to embed that as part of a DLT pipeline or is the recommendation to include it as part of the transformation step?
[ Parent | on HN ]
Re:Show HN: Data load tool(dlt)-Python library to automate t... by MatthausK on Wednesday October 25, 2023 @06:14AM (#38011214)
You can use pydantic models to define schemas, validate data (we also load instances of the models natively): https://dlthub.com/docs/general-usage/resource#define-a-sche...

We have a PR (https://github.com/dlt-hub/dlt/pull/594) that is about to merge that makes the above highly configurable, between evolution and hard stopping: - you will be able to totally freeze schema and reject bad rows - or accept the data for existing columns but not new columns - or accept some fields based on rules'

[ Parent | on HN ]