News for hackers. Stuff that matters.
Re:Dat: non-profit, secure, and distributed package manager ... by jhoechtl on Wednesday September 27, 2017 @12:53AM (#15345017)
The git toolset is, out of the box, weak on tabular data. Also out of the box large file support requires one central repository.

None of that is true for Dat

Re:Dat: non-profit, secure, and distributed package manager ... by icebraining on Wednesday September 27, 2017 @02:09AM (#15345284)
out of the box large file support requires one central repository.

How so? As far as I know, git doesn't treat large files any differently than small ones.

Re:Dat: non-profit, secure, and distributed package manager ... by watson on Wednesday September 27, 2017 @03:47AM (#15345662)
I'll try to answer your question, but it's been a while since I last looked at it, so I might get some/most of this wrong - so don't shoot me:

As far as I know git isn't good at storing binary data. Git depends on line breaks to be able to diff and make change sets. If you store a binary file in git and make an update to it - even though that update only changed 1 byte, the entire new version of the file is stored again. Dat uses Rabin fingerprinting to intelligently slice binary files into chunks that are less likely to change. That make dat a lot more efficient at storing, versioning, and syncing videos, images, and other large binary files.

Re:Dat: non-profit, secure, and distributed package manager ... by icebraining on Wednesday September 27, 2017 @04:53AM (#15345981)
Conceptually, git doesn't use change-sets, each commit is a snapshot of all the files in the current version. For storage and transmission efficiency, though, it can store and send them in packfiles, which use delta compression based on LibXDiff - which uses Rabin's fingerprint algorithm (as well as other algorithm by Joshua P. MacDonald) for binary files.
Re:Dat: non-profit, secure, and distributed package manager ... by rspeer on Wednesday September 27, 2017 @03:49AM (#15345674)
GitHub definitely treats large files differently from small files (it rejects them), and a Git repository you can't push to GitHub is very different from a typical Git repository.

Other than that, Git treating large files the same as small ones is part of the problem, and the reason that centralized extensions exist. You wouldn't want "git clone" to clone the complete history of every large file, while that's not a concern at all for small files.

Re:Dat: non-profit, secure, and distributed package manager ... by icebraining on Wednesday September 27, 2017 @04:57AM (#15345999)
a Git repository you can't push to GitHub is very different from a typical Git repository.

When comparing to Dat, I don't see how; you can't push any Dat repository to Github.

Other than that, Git treating large files the same as small ones is part of the problem, and the reason that centralized extensions exist. You wouldn't want "git clone" to clone the complete history of every large file, while that's not a concern at all for small files.

You don't need centralized extensions for that, though. I use git-annex, which is completely P2P.

Re:Dat: non-profit, secure, and distributed package manager ... by rspeer on Wednesday September 27, 2017 @12:19PM (#15349314)
I was responding to the claim that "git doesn't treat large files any differently than small ones". I was not saying anything about dat or git-annex. Now it's clear that you are treating your large files differently from small ones.

Good luck with git-annex.

Re:Dat: non-profit, secure, and distributed package manager ... by icebraining on Wednesday September 27, 2017 @01:39PM (#15350251)
Right, but the claim was about git and you responded with Github, which is a different thing.
Re:Dat: non-profit, secure, and distributed package manager ... by erikb on Wednesday September 27, 2017 @03:45AM (#15345656)
So, git is not good, but Dat isn't good either?