News for hackers. Stuff that matters.
Re:DVC – Open Source Machine Learning Version Control System by cyphar on Sunday February 10, 2019 @08:36PM (#19131435)
> Correct, there are no data diffs. DVC just stores blobs and you can GC the old ones

Have you looked into using content-defined chunking (a-la restic or borgbackup) so that you get deduplication without the need to send around diffs? This is related to a problem that I'm working on solving in OCI (container) images[1].

[1]: https://www.cyphar.com/blog/post/20190121-ociv2-images-i-tar

Re:DVC – Open Source Machine Learning Version Control System by dmpetrov on Monday February 11, 2019 @08:06AM (#19134127)
Content-defined chunks - very interesting. I'd suggest you ask this question in DVC issue tracker or DVC channel https://dvc.org/chat