News for hackers. Stuff that matters.
Re:Dat – A git-like tool for large datasets by derefr on Saturday April 26, 2014 @12:04AM (#7650399)
I've been pondering hacking on git to give it a pluggable object-storage API. That way, you could have one repo dump its objects into a Fossil file-system server, or an S3 bucket, or a DHT, and then, when another repo fetched from it, it'd just be pulling metadata (branch-heads and tags), and getting the actual data from the object-storage instead. I'm pretty sure this'd fix the "git can't store large files" problem, especially if you also allowed composite file objects, and perhaps per-object concurrent BitTorrent retrieval.
Re:Dat – A git-like tool for large datasets by beliu on Saturday April 26, 2014 @01:46AM (#7650567)
See also: https://camlistore.org/ (unlike Dat, doesn't target datasets, but more generic user data, like photos and files; more Dropbox than npm). It has a pretty cool blob model, exposes a FUSE interface, has a slick photo-browsing UI, and integrates with S3 and (I think) Glacier.
Re:Dat – A git-like tool for large datasets by sciurus on Saturday April 26, 2014 @01:04PM (#7652043)
For a recent look at camlistore see http://lwn.net/SubscriberLink/595746/2adfdecdb0de2577/

(and if you like that article, consider subscribing to LWN!)

Re:Dat – A git-like tool for large datasets by rpedela on Saturday April 26, 2014 @12:11AM (#7650418)
Isn't that what git-annex is for?

https://git-annex.branchable.com/

Re:Dat – A git-like tool for large datasets by derefr on Saturday April 26, 2014 @01:30AM (#7650541)
git-annex is an ugly hack to, basically, reimplement git-with-this-feature on top of git. What I'm suggesting would be to reimplement git itself on top of distributed object storage -- effectively, to split git into two tools:

1. one that commits treeish diffs into objects, and then stuffs them into arbitrary content-addressable storage;

2. and one that manipulates (local or remote) mutable tables of refs to content-hash-URNs -- or "git repositories" for short.

The key component would be a facility (either a library or an OS service) for resolving content-hash-URNs into file-descriptors in some content-addressable storage system. The critical point is that this would not be a git component -- the committing tool wouldn't be in charge of where its objects go, and the repository tool wouldn't be in charge of resolving their URNs. Instead, both would just be relying on the host to decide how to get objects from/put objects into storage, and it's the host that would have a distribution strategy set up for all of its content-addressable objects, not just the ones that happen to be attached to git.

Re:Dat – A git-like tool for large datasets by platz on Saturday April 26, 2014 @01:21AM (#7650531)
Also some interesting related projects on https://git-annex.branchable.com/not/
Re:Dat – A git-like tool for large datasets by espeed on Saturday April 26, 2014 @12:52AM (#7650485)
Re:Dat – A git-like tool for large datasets by beagle3 on Saturday April 26, 2014 @08:10AM (#7651128)
Libgit2 does this internally, although it doesn't expose the underlying store (so you wouldn't know to use S3 even if you could get objects from there)
Re:Dat – A git-like tool for large datasets by michaelmior on Saturday April 26, 2014 @12:14AM (#7650428)
You would end up with pretty huge latency for a lot of common operations in that case. Every checkout, diff, and status for example.
Re:Dat – A git-like tool for large datasets by derefr on Saturday April 26, 2014 @01:12AM (#7650515)
That depends on how replicated/distributed the object storage is. In a DHT scenario, you might already be storing some of the objects yourself before you've even requested them.

The remote server can obviously be running an object-storage node itself, and so can your own local git repo -- the point is that you can have objects in more places than just your machine and the repo it's pulling from.

Re:Dat – A git-like tool for large datasets by zobzu on Saturday April 26, 2014 @12:20AM (#7650432)
yeah it sounds like svn all over again
Re:Dat – A git-like tool for large datasets by TillE on Saturday April 26, 2014 @04:30AM (#7650778)
You could probably modify Mercurial's largefiles extension to use S3, etc as a central store.
Re:Dat – A git-like tool for large datasets by addisaden on Saturday April 26, 2014 @03:14AM (#7650681)
just plain git with also large datafiles triggers my mind a lot. not databases ... just all kinds of data. (like large pdfs) ..