News for hackers. Stuff that matters.
Re:Unpacking Git packfiles by chimeracoder on Wednesday June 24, 2015 @03:07PM (#9773763)
Author here. I discovered this while working on a clean-room implementation of Git in pure Go. While there are a lot of references to packfiles online, surprisingly, the actual format of packfiles was rather underdocumented. Most resources just mention that they exist, and describe how to use `git verify-pack` to inspect a packfile, without explaining how to parse packfiles and apply deltas.

I decided to write this up to save others the trouble of having to reverse-engineer it from scratch!

Re:Unpacking Git packfiles by andmarios on Wednesday June 24, 2015 @03:22PM (#9773869)
Just wanted to say that your article is very nicely written. It is easy to follow and keeps the reader interested. Kudos. :)
Re:Unpacking Git packfiles by bluehawk on Wednesday June 24, 2015 @07:09PM (#9775059)
Super minor nitpick:

"Since 0xFF (256) is the largest value that can fit into a single byte"

0xFF is 255 not 256

Re:Unpacking Git packfiles by wereHamster on Wednesday June 24, 2015 @03:29PM (#9773910)
Re:Unpacking Git packfiles by chimeracoder on Wednesday June 24, 2015 @03:33PM (#9773938)
Yes, I link to a different version of that same file in the article (my link points to the version hosted on kernel.org, rather than Github). It provides a bit of high-level context, but by itself it doesn't provide enough detail to actually reimplement the corresponding Git functions.

Aside from being more terse and (IMHO) more difficult to read than prose with examples and non-ASCII diagrams, that file doesn't explain the context and motivation for packfiles, and it doesn't cover the parsing and application of deltas at all.

Re:Unpacking Git packfiles by ethomson on Wednesday June 24, 2015 @03:40PM (#9773986)
If you found that piece of documentation deficient while implementing a packfile parser, then it would be nice to update it to include those details that were lacking to help the next person to reimplement git.
Re:Unpacking Git packfiles by chimeracoder on Wednesday June 24, 2015 @03:44PM (#9774010)
Thanks for the suggestion! I'll take a look at the contribution process for the Git project.
Re:Unpacking Git packfiles by Dunnorandom on Wednesday June 24, 2015 @05:10PM (#9774503)
Great article! Just one small correction: OFS_REF_DELTA should be OBJ_REF_DELTA.
Re:Unpacking Git packfiles by JoshTriplett on Wednesday June 24, 2015 @04:00PM (#9774106)
What was your motivation to implement from scratch, rather than using libgit2?
Re:Unpacking Git packfiles by chimeracoder on Wednesday June 24, 2015 @04:19PM (#9774208)
A combination of the following:

- Cross-compilation (trivial in Go, less so in C)

- A chance to learn about the really dark, thorny corners of Git

And for what it's worth, source control is Git's intended use case, but people do use it for other purposes as well (like managing personal media collections across multiple devices[0]). Git has become a protocol or a platform in addition to a VCS[1].

But there aren't very many FOSS clean-room implementations of Git, at least not this far down the chain (packfiles). One of the best ways to discover hidden implementation issues or oversights in a spec or existing documentation is to try and reimplement it, which has the effect of strengthening the platform itself in the long run.

[0] e.g. https://git-annex.branchable.com/

[1] Bitcoin is a bit of a hot-button topic, but it's similar to bitcoin in this regard: the tool itself is intended for financial transactions, but people have already started to use it for all sorts of unrelated use cases.

Re:Unpacking Git packfiles by rakoo on Wednesday June 24, 2015 @05:27PM (#9774610)
> But there aren't very many FOSS clean-room implementations of Git, at least not this far down the chain (packfiles).

I know at least of gogit (https://github.com/speedata/gogit), if only a little, because I've contributed once to it. I don't want to belittle your project, but I'd like to know: what are the differences between gitgo and gogit ?

(I see at least one similarity: the name is extraordinarily unimaginative :)

Re:Unpacking Git packfiles by bicolao on Wednesday June 24, 2015 @07:17PM (#9775094)
> One of the best ways to discover hidden implementation issues or oversights in a spec or existing documentation...

There are issues. And there is the legendary "pack v4" (current pack versions are 2 or 3) but it still a work in progress.

Re:Unpacking Git packfiles by nathancahill on Wednesday June 24, 2015 @04:36PM (#9774300)
Kudos. Just noting that one of libgit2's primary strengths is 100% cross-platform portability. Is this different from your cross-compilation goal? I'm not familiar with Go.
Re:Unpacking Git packfiles by giovannibajo1 on Wednesday June 24, 2015 @05:29PM (#9774621)
Yes because Go generates static binaries and requires no toolchain. Once you link a C library, you loose these benefits and cross-compiling returns to the "normal" difficulty. In addition to that, Go code may be buggy but it's safe, while linking a C library means dropping this guarantee.

So in the Go ecosystem it is actually preferred to use pure Go libraries not to loose these benefits.

Re:Unpacking Git packfiles by ngoldbaum on Wednesday June 24, 2015 @06:14PM (#9774839)
Is it really a clean-room implementation (in the GPL-enforcement sense) if you're looking at the git internal documentation?
Re:Unpacking Git packfiles by belorn on Thursday June 25, 2015 @04:19AM (#9776571)
I doubt it. A clean-room implementation is intended to demonstrate to a court that no knowledge of the copyrighted material was available to the implementers during the creation of the work. Any similarities to the "original" work must then be caused by functional constraints resulting from compatibility requirements, thus free from creative elements (not protected by copyright).

If the internal documentation include any creative elements which are reflected by the git implementation, then the clean-room would be contaminated.

Re:Unpacking Git packfiles by eru on Wednesday June 24, 2015 @09:02PM (#9775402)
Interesting legal question. I'd bet on the `yes' side, but don't have any arguments.