Re:Show HN: A versioned filesystem inspired by Git
by tolmasky on Tuesday August 28, 2012 @12:23PM
(#4444472)
OK, first off -- awesome! Secondly, I have a number of questions that probably have to do with the fact that I'm unfamiliar with a lot of this stuff.
1. Would it be possible to build this fully on top of git (perhaps using libgit2)? I ask, because the "holy" grail for me is finding some magic way for my designers to use git without knowing it. Right now my designers (and I think most non-programmer people), love to use Dropbox. Dropbox has a number of issues as a shared project tool in my perspective (not the least of which being that you are only allowed ONE dropbox account per computer, so you start having a shared folder mess and your dropbox begins to balloon in size on every computer with that account). It appears that something like this could look like Dropbox to everyone disinterested in vcs and act like Git to everyone else who cares.
2. The other reason for libgit2 is to ideally also git-push after every edit/save, to something like github.
Re:Show HN: A versioned filesystem inspired by Git
by artagnon on Tuesday August 28, 2012 @12:38PM
(#4444550)
1. I only use (a modified version of) packfile/ packfile idx from Git. No commits are actually created. It sounds like you're looking for something like flashbake (http://bitbucketlabs.net/flashbake/).
2. phoenixfs doesn't create commits, so no git-push.
3. I currently don't have access to an OS X box, sorry :(
Re:Show HN: A versioned filesystem inspired by Git
by s_tec on Tuesday August 28, 2012 @03:04PM
(#4445300)
Have you tried SparkleShare? It sounds exactly like what you want. It connects to a normal Git server and presents the repository as a Dropbox-esque folder. It has support for Linux, Windows, and Mac, although it's mainly a Linux project.
Re:Show HN: A versioned filesystem inspired by Git
by sciurus on Tuesday August 28, 2012 @06:01PM
(#4446109)
More info:
"SparkleShare is a 100% free and open source, cross-platform file sharing system started by Hylke Bons. The idea for the project came from the 2010 GNOME UX Hackfest. It's very similar to Dropbox or Box.net, but it is backed by the free and open source version control system git.
Basically, the way it works is that you will have a special folder on your computer, called 'Sparkleshare', and everything in that folder is also present in a version-controlled remote git repository. If you create a new file in a Sparkleshare folder, it will immediately be committed and pushed into the remote git repository, and everyone else who is using Sparkleshare to connect to that folder will be able to see the file you just created appear in their Sparkleshare folders. If someone else opens the file you just created and modifies it, their Sparkleshare client will automatically commit and push their changes to the repository, and you'll be able to see them in your Sparkleshare folder (and you'll get a handy notification!)
Sparkleshare is a great, free & open source way to collaborate on files, without worrying about manual version control or having to manually upload and broadcast links to your work."
Re:Show HN: A versioned filesystem inspired by Git
by tolmasky on Tuesday August 28, 2012 @12:35PM
(#4444534)
Yeah I asked for an invite ages ago, haven't heard back. if they ever let me in it might do the trick.
Separately, actual github for designers would be cooler because I could:
1. Have an "artwork" git repo which my designers interact with in this way, but which I include as a submodule to a larger git repo for my programmers.
2. Use the same bug tracking for art as I do programming
3. Backup the git repo because its an open protocol vs. some proprietary thing.
Again though, I'm lazy and don't have time to write project management tools for myself so if these guys ever got around to sending me an invite I'd probably just settle.
Re:Show HN: A versioned filesystem inspired by Git
by fuzzix on Tuesday August 28, 2012 @11:15AM
(#4444081)
I like it. It reminds me of VMS's versioning.
I implemented something like VMS style versioning with Fuse and Perl for a talk/demo a couple of years ago - it simply created a new version on open() if any writeable flags were passed - not robust, but functional enough to demo.
Re:Show HN: A versioned filesystem inspired by Git
by spartango on Tuesday August 28, 2012 @05:30PM
(#4445978)
This is true, but not without drawbacks:
* Using FUSE you gain the atomicity through synchronicity; when a write occurs, you stand between the disk and request. In so doing, you introduce the potential for latency (nevermind the latency inherent to FUSE).
* Using fsevents, you get notifications asynchronously, so there's no I/O performance hit. You may not get the events right away, but with fsevent APIs there's a bounded queue so you generally don't miss events, unless there are too many and the buffer is overflowed[1,2]. The events don't contain the actual file data, so you will lose intermediate edit data.
Thinking about this, it would be cool to exploit a Copy-on-Write filesystem (like ZFS) to scalably add the edit data to fsevent apis. I suppose you could preserve write buffers in memory temporarily instead of CoW, but that would have a nasty memory footprint.
Re:Show HN: A versioned filesystem inspired by Git
by p9idf on Tuesday August 28, 2012 @12:45PM
(#4444587)
Does your file system create revisions of the entire tree, or only revisions of individual files within the tree as VMS and RCS do?† Because there are some problems with the latter approach. One problem described by Geoff Collyer (the C News guy) is that "version numbers of related files are not kept in synch, so you can't just cp *';6' /tmp/compile".††
† I tried answering that question myself, but your program kept crashing.
Re:Show HN: A versioned filesystem inspired by Git
by gcr on Tuesday August 28, 2012 @01:31PM
(#4444858)
Can this handle large binary files? That's been my main problem with storing backups of my servers in git, and it's one of the strong-ish points of git-annex.
There seems to be nothing inherently limiting in Git's repository design. Interested parties can look into bup, for example, which stores large binary files in git's packfile format. https://github.com/apenwarr/bup
Re:Show HN: A versioned filesystem inspired by Git
by artagnon on Tuesday August 28, 2012 @10:34AM
(#4443820)
It's a bit of an overkill for my purposes- I've only copied out (and modified) a little of the packfile/ packfile index generation code from git.git. Plus, dealing with the actual code is a great learning exercise :)
Re:Show HN: A versioned filesystem inspired by Git
by josephscott on Tuesday August 28, 2012 @01:10PM
(#4444744)
Agreed, VMS filesystem was the first thing that came to mind for me as well. Seems rather sad that in 2012 we don't have that as a standard feature like we did back then.
For the record, you don't have to be all that old to have used VMS's filesystem :-)
Re:Show HN: A versioned filesystem inspired by Git
by nicholassmith on Tuesday August 28, 2012 @10:14AM
(#4443739)
This looks pretty good. I had a look at doing something similar (but wayyyyy smaller scale than full FS), and gave up as there was many dragons. Nice to see someone pull it off.
Re:Show HN: A versioned filesystem inspired by Git
by artagnon on Tuesday August 28, 2012 @12:32PM
(#4444519)
There's a FUSE for OS X, so a port should be possible (should be possible for other *nix too, as long as they have FUSE). I currently don't have access to an OS X machine, so no immediate plans.
1. Would it be possible to build this fully on top of git (perhaps using libgit2)? I ask, because the "holy" grail for me is finding some magic way for my designers to use git without knowing it. Right now my designers (and I think most non-programmer people), love to use Dropbox. Dropbox has a number of issues as a shared project tool in my perspective (not the least of which being that you are only allowed ONE dropbox account per computer, so you start having a shared folder mess and your dropbox begins to balloon in size on every computer with that account). It appears that something like this could look like Dropbox to everyone disinterested in vcs and act like Git to everyone else who cares.
2. The other reason for libgit2 is to ideally also git-push after every edit/save, to something like github.
3. Max OS X support please :)
2. phoenixfs doesn't create commits, so no git-push.
3. I currently don't have access to an OS X box, sorry :(
"SparkleShare is a 100% free and open source, cross-platform file sharing system started by Hylke Bons. The idea for the project came from the 2010 GNOME UX Hackfest. It's very similar to Dropbox or Box.net, but it is backed by the free and open source version control system git.
Basically, the way it works is that you will have a special folder on your computer, called 'Sparkleshare', and everything in that folder is also present in a version-controlled remote git repository. If you create a new file in a Sparkleshare folder, it will immediately be committed and pushed into the remote git repository, and everyone else who is using Sparkleshare to connect to that folder will be able to see the file you just created appear in their Sparkleshare folders. If someone else opens the file you just created and modifies it, their Sparkleshare client will automatically commit and push their changes to the repository, and you'll be able to see them in your Sparkleshare folder (and you'll get a handy notification!)
Sparkleshare is a great, free & open source way to collaborate on files, without worrying about manual version control or having to manually upload and broadcast links to your work."
http://sparkleshare.org/
Separately, actual github for designers would be cooler because I could:
1. Have an "artwork" git repo which my designers interact with in this way, but which I include as a submodule to a larger git repo for my programmers.
2. Use the same bug tracking for art as I do programming
3. Backup the git repo because its an open protocol vs. some proprietary thing.
Again though, I'm lazy and don't have time to write project management tools for myself so if these guys ever got around to sending me an invite I'd probably just settle.
With that being said, I use SourceTree[1] every day and love it.
[1]http://www.sourcetreeapp.com/
Any advantage of using a full-scale FS over something like git-annex (http://git-annex.branchable.com/)?
I implemented something like VMS style versioning with Fuse and Perl for a talk/demo a couple of years ago - it simply created a new version on open() if any writeable flags were passed - not robust, but functional enough to demo.
edit Possessed by VMS, not multiple VMSes.
I'm not the biggest fan of FUSE filesystems(until highly-tuned, ala NTFS-3G), but they certainly do work.
* Using FUSE you gain the atomicity through synchronicity; when a write occurs, you stand between the disk and request. In so doing, you introduce the potential for latency (nevermind the latency inherent to FUSE).
* Using fsevents, you get notifications asynchronously, so there's no I/O performance hit. You may not get the events right away, but with fsevent APIs there's a bounded queue so you generally don't miss events, unless there are too many and the buffer is overflowed[1,2]. The events don't contain the actual file data, so you will lose intermediate edit data.
Thinking about this, it would be cool to exploit a Copy-on-Write filesystem (like ZFS) to scalably add the edit data to fsevent apis. I suppose you could preserve write buffers in memory temporarily instead of CoW, but that would have a nasty memory footprint.
[1]http://msdn.microsoft.com/en-us/library/system.io.filesystem... - Internal buffer size
[2]http://docs.oracle.com/javase/7/docs/api/java/nio/file/Watch... (see the watchkey queue and overflow)
† I tried answering that question myself, but your program kept crashing.
†† http://9fans.net/archive/2001/11/759
† Fixed. Sorry about that.
There seems to be nothing inherently limiting in Git's repository design. Interested parties can look into bup, for example, which stores large binary files in git's packfile format. https://github.com/apenwarr/bup
http://www.hpl.hp.com/personal/alistair_veitch/papers/elepha...
If only half the stuff done by the likes of IBM and HP were to make it into common usage the world would be a very different place.
Also: I'm old enough to have used VMS's filesystem. :(
For the record, you don't have to be all that old to have used VMS's filesystem :-)
(† Apparently: http://en.wikipedia.org/wiki/Versioning_file_system)
[1]: http://en.wikipedia.org/wiki/ZFS#Snapshots_and_clones
There's also git bup[2] and git-annex[3].
[1] http://www.dedoimedo.com/computers/btrfs-snapshots.html
[2] https://github.com/apenwarr/bup
[3] http://git-annex.branchable.com/
https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L59...
Missed malloc check.
https://github.com/artagnon/phoenixfs/blob/master/fuse.c#L82
system calls like mkdir can fail. Better to check return values of such calls.
This might be what I'm thinking of: http://joeyh.name/code/etckeeper/
Any plans for MacOS (or other *nix) support?