Re:DataChain: DBT for Unstructured Data
by dmpetrov on Monday November 04, 2024 @02:44PM
(#42045331)
Exactly! DataChain does lazy compute. It will read metadata/json while applying filtering and only download a sample of data files (jpg) based on the filter.
This way, you might end up downloading just 1% of your data, as defined by the metadata filter.
This way, you might end up downloading just 1% of your data, as defined by the metadata filter.