DVC – Open Source Machine Learning Version Control System

Re:DVC – Open Source Machine Learning Version Control System by mroutis on Sunday February 10, 2019 @04:46PM (#19130256)

Git itself is not suitable for huge files:

- Large binary files tend to be not very "deflatable"

- xdelta (used in Git to diff files) tries to load the entire content of a file into memory, at once.

This is why there are solutions like Git-LFS, where you keep your versions on a remote server / cloud storage and you use git to track only the "metadata" files.

DVC implemented its own solution, in order to be SCM agnostic and cloud flexible (supporting different remote storages).

Here's more info comparing DVC to similar/related technologies: https://dvc.org/doc/dvc-philosophy/related-technologies

EDIT: formatting

[ Parent | on HN ]

Re:DVC – Open Source Machine Learning Version Control System by codetrotter on Monday February 11, 2019 @05:20AM (#19133336)

Thank you for the link, that’s the kind of comparison I was looking for all the way down to even talking about how DVC compares to git-annex :)

[ Parent | on HN ]