News for hackers. Stuff that matters.
Re:Data Version Control by ajoseps on Saturday October 19, 2024 @05:31PM (#41890932)
if the data files are all just text files, what are the differences between DVC and using plain git?
Re:Data Version Control by miki123211 on Saturday October 19, 2024 @05:58PM (#41891080)
DVC does a lot more than git.

It essentially makes sure that your results can reproducibly be generated from your original data. If any script or data file is changed, the parts of your pipeline that depend on it, possibly recursively, get re-run and the relevant results get updated automatically.

There's no chance of e.g. changing the structure of your original dataset slightly, forgetting to regenerate one of the intermediate models by accident, not noticing that the script to regenerate it doesn't work any more due to the new dataset structure, and then getting reminded a year later when moving to a new computer and trying to regen everything from scratch.

It's a lot like Unix make, but with the ability to keep track of different git branches and the data / intermediates they need, which saves you from needing to regen everything every time you make a new checkout, lets you easily exchange large datasets with teammates etc.

In theory, you could store everything in git, but then every time you made a small change to your scripts that e.g. changed the way some model works and slightly adjusted a score for each of ten million rows, your diff would be 10m LOC, and all versions of that dataset would be stored in your repo, forever, making it unbelievably large.

Re:Data Version Control by azinman2 on Saturday October 19, 2024 @07:52PM (#41891756)
So where do the adjusted 10M rows live instead? S3?
Re:Data Version Control by thangngoc89 on Saturday October 19, 2024 @10:55PM (#41892535)
DVC support multiple remotes. S3 is one of them, there are also WebDAV, local FS, Google Drive, and a bunch of others. You could see the full list here [0]. Disclaimer: not affiliated with DVC in anyway, just a user.

[0] https://dvc.org/doc/user-guide/data-management/remote-storag...

Re:Data Version Control by woodglyst on Sunday October 20, 2024 @08:20AM (#41894861)
This sounds a lot like the experimental project Jacquard [0] from Ink & Switch.

[0] https://www.inkandswitch.com/jacquard/notebook/

Re:Data Version Control by amelius on Sunday October 20, 2024 @09:32AM (#41895262)
Sounds like it is more a framework than a tool.

Not everybody wants a framework.

Re:Data Version Control by JadeNB on Sunday October 20, 2024 @11:20AM (#41895874)
> Sounds like it is more a framework than a tool.

> Not everybody wants a framework.

The second part of this comment seems strange to me. Surely nothing on Hacker News is shared with the expectation that it will be interesting, or useful, to everyone. Equally, surely there are some people on HN who will be interested in a framework, even if it might be too heavy for other people.

Re:Data Version Control by amelius on Sunday October 20, 2024 @12:05PM (#41896274)
Just saying that what makes Git so appealing is that it does one thing well, and from this view DVC seems to be in an entirely different category.
Re:Data Version Control by stochastastic on Sunday October 20, 2024 @01:17PM (#41896912)
It doesn’t force you to use any of the extra functionality. My team has been using it just for the version control part for a couple years and it has worked great.
Re:Data Version Control by bach4ants on Saturday October 26, 2024 @08:33AM (#41954372)
Yep. I personally like DVC's pipeline implementation because it's lightweight and language-agnostic, but haven't gotten into using their experiment tracking features.
Re:Data Version Control by dmpetrov on Saturday October 19, 2024 @05:55PM (#41891059)
In this cases, you need DVC if:

1. File are too large for Git and Git LFS.

2. You prefer using S3/GCS/Azure as a storage.

3. You need to track transformations/piplines on the file - clean up text file, train mode, etc.

Otherwise, vanilla Git may be sufficient.

Re:Data Version Control by agile-gift0262 on Sunday October 20, 2024 @03:03AM (#41893500)
It's not just to manage file versioning. Yo can define a pipeline with different stages, the dependencies and outputs of each stage and DVC will figure out which stages need running depending on what dependencies have changed. Stages can also output metrics and plots, and DVC has utilities to expose, explore and compare those.