News for hackers. Stuff that matters.
Re:Show HN: Data load tool(dlt)-Python library to automate t... by nerdponx on Wednesday October 25, 2023 @04:52PM (#38018224)
Fair enough, especially if explicit alerting is involved.

Have you considered a hybrid solution, something that generates a contract from a large corpus of data, which can then be deployed statically?

I consider "responding to change" as a somewhat different scenario from "heterogeneous but not changing". So statically generating a contract from an existing corpus supports the latter.

I could also envision some kind of graceful degradation, where you have a static contract, but you have dynamic adjustments instead of outright failures if the data does not conform to that contract.

Re:Show HN: Data load tool(dlt)-Python library to automate t... by vasa_ on Wednesday November 01, 2023 @01:40PM (#38101882)
I worked with dlt guys on exactly that. Using OpenAI functions to generate a schema for the data based on the raw data structure. You can check that work here: https://github.com/topoteretes/PromethAI-Memory It's in the level 1 folder