Every time I see DVC mentioned I always feel like the idea was so close (and perhaps right in intuition to use git for everything) but the execution had just enough friction that I looked elsewhere. Small DX improvements really do cascade pretty far.
But for a data project it would be a big pain to have separate worktrees just to work around what IMO is a usage anti-pattern to begin with!
[1] https://iterative.ai/blog/experiment-refs
[2] https://marketplace.visualstudio.com/items?itemName=Iterativ...
It's neat for research as it stores the data on scientific data repositories like Zenodo and you get DOIs.
A relational database like Dolt seems like a better fit. You want to be able to query by experiment name, date, test results, and other metadata.
Let me know if I'm doing it wrong! What's your use case?
Out of the box you can't. We're taking a different approach at work (XetHub). GitHub sees pointer files but we embed rendered views and diffs (supporting more file types incrementally) using a Github app from our service.
https://www.pachyderm.com/blog/data-versioning-comparing-dvc... https://www.dolthub.com/blog/2022-04-27-data-version-control...
As far as I know, DVC is better than Pachyderm for small datasets, but Pachyderm scales way better
I think it's a shame it took Airflow and similar several more years to realize "each step is a docker container" is the right way to build a dag. It's not clear to me why Pachyderm was left behind while Prefect and Dagster became serious contenders, and Airflow/Astronomer started recommending everyone use it just like Pachyderm (container per step).
HF seems like "GitHub for models and datasets", it has a cool brand and everyone in ML uses it in some capacity. But when it comes to _company_ needs like private datasets/models, experiment tracking, CI integrations, etc. it seems WandB is a superset of HF.
HF has an enterprise offering, but it seems to be de-prioritized, and I think you'd still need WandB or MLFlow for experiment tracking?