Re:Show HN: Data load tool(dlt)-Python library to automate t...
by vasa_ on Wednesday November 01, 2023 @01:40PM
(#38101882)
I worked with dlt guys on exactly that. Using OpenAI functions to generate a schema for the data based on the raw data structure. You can check that work here: https://github.com/topoteretes/PromethAI-Memory
It's in the level 1 folder
Have you considered a hybrid solution, something that generates a contract from a large corpus of data, which can then be deployed statically?
I consider "responding to change" as a somewhat different scenario from "heterogeneous but not changing". So statically generating a contract from an existing corpus supports the latter.
I could also envision some kind of graceful degradation, where you have a static contract, but you have dynamic adjustments instead of outright failures if the data does not conform to that contract.