Re:Unpacking Git packfiles
by chimeracoder on Wednesday June 24, 2015 @03:07PM
(#9773763)
Author here. I discovered this while working on a clean-room implementation of Git in pure Go. While there are a lot of references to packfiles online, surprisingly, the actual format of packfiles was rather underdocumented. Most resources just mention that they exist, and describe how to use `git verify-pack` to inspect a packfile, without explaining how to parse packfiles and apply deltas.
I decided to write this up to save others the trouble of having to reverse-engineer it from scratch!
Re:Unpacking Git packfiles
by chimeracoder on Wednesday June 24, 2015 @03:33PM
(#9773938)
Yes, I link to a different version of that same file in the article (my link points to the version hosted on kernel.org, rather than Github). It provides a bit of high-level context, but by itself it doesn't provide enough detail to actually reimplement the corresponding Git functions.
Aside from being more terse and (IMHO) more difficult to read than prose with examples and non-ASCII diagrams, that file doesn't explain the context and motivation for packfiles, and it doesn't cover the parsing and application of deltas at all.
Re:Unpacking Git packfiles
by ethomson on Wednesday June 24, 2015 @03:40PM
(#9773986)
If you found that piece of documentation deficient while implementing a packfile parser, then it would be nice to update it to include those details that were lacking to help the next person to reimplement git.
Re:Unpacking Git packfiles
by chimeracoder on Wednesday June 24, 2015 @04:19PM
(#9774208)
A combination of the following:
- Cross-compilation (trivial in Go, less so in C)
- A chance to learn about the really dark, thorny corners of Git
And for what it's worth, source control is Git's intended use case, but people do use it for other purposes as well (like managing personal media collections across multiple devices[0]). Git has become a protocol or a platform in addition to a VCS[1].
But there aren't very many FOSS clean-room implementations of Git, at least not this far down the chain (packfiles). One of the best ways to discover hidden implementation issues or oversights in a spec or existing documentation is to try and reimplement it, which has the effect of strengthening the platform itself in the long run.
[1] Bitcoin is a bit of a hot-button topic, but it's similar to bitcoin in this regard: the tool itself is intended for financial transactions, but people have already started to use it for all sorts of unrelated use cases.
Re:Unpacking Git packfiles
by rakoo on Wednesday June 24, 2015 @05:27PM
(#9774610)
> But there aren't very many FOSS clean-room implementations of Git, at least not this far down the chain (packfiles).
I know at least of gogit (https://github.com/speedata/gogit), if only a little, because I've contributed once to it. I don't want to belittle your project, but I'd like to know: what are the differences between gitgo and gogit ?
(I see at least one similarity: the name is extraordinarily unimaginative :)
Re:Unpacking Git packfiles
by nathancahill on Wednesday June 24, 2015 @04:36PM
(#9774300)
Kudos. Just noting that one of libgit2's primary strengths is 100% cross-platform portability. Is this different from your cross-compilation goal? I'm not familiar with Go.
Re:Unpacking Git packfiles
by giovannibajo1 on Wednesday June 24, 2015 @05:29PM
(#9774621)
Yes because Go generates static binaries and requires no toolchain. Once you link a C library, you loose these benefits and cross-compiling returns to the "normal" difficulty. In addition to that, Go code may be buggy but it's safe, while linking a C library means dropping this guarantee.
So in the Go ecosystem it is actually preferred to use pure Go libraries not to loose these benefits.
Re:Unpacking Git packfiles
by belorn on Thursday June 25, 2015 @04:19AM
(#9776571)
I doubt it. A clean-room implementation is intended to demonstrate to a court that no knowledge of the copyrighted material was available to the implementers during the creation of the work. Any similarities to the "original" work must then be caused by functional constraints resulting from compatibility requirements, thus free from creative elements (not protected by copyright).
If the internal documentation include any creative elements which are reflected by the git implementation, then the clean-room would be contaminated.
Re:Unpacking Git packfiles
by lucb1e on Wednesday June 24, 2015 @05:15PM
(#9774536)
This ("Unpacking Git packfiles") was a CTF challenge a few weeks ago (at the Haxpo CTF in Amsterdam), except we weren't given the original repository, we only got a pcap dump of the traffic. Using `git extract-objects` I was able to unpack them into object files (stored in .git/objects/xx/*) but even these were not readable. Eventually found some zpipe command that did the trick. What a pain to do this with common tooling if you don't have the time to dive into the format and write a real unpacker.
I decided to write this up to save others the trouble of having to reverse-engineer it from scratch!
"Since 0xFF (256) is the largest value that can fit into a single byte"
0xFF is 255 not 256
Aside from being more terse and (IMHO) more difficult to read than prose with examples and non-ASCII diagrams, that file doesn't explain the context and motivation for packfiles, and it doesn't cover the parsing and application of deltas at all.
- Cross-compilation (trivial in Go, less so in C)
- A chance to learn about the really dark, thorny corners of Git
And for what it's worth, source control is Git's intended use case, but people do use it for other purposes as well (like managing personal media collections across multiple devices[0]). Git has become a protocol or a platform in addition to a VCS[1].
But there aren't very many FOSS clean-room implementations of Git, at least not this far down the chain (packfiles). One of the best ways to discover hidden implementation issues or oversights in a spec or existing documentation is to try and reimplement it, which has the effect of strengthening the platform itself in the long run.
[0] e.g. https://git-annex.branchable.com/
[1] Bitcoin is a bit of a hot-button topic, but it's similar to bitcoin in this regard: the tool itself is intended for financial transactions, but people have already started to use it for all sorts of unrelated use cases.
I know at least of gogit (https://github.com/speedata/gogit), if only a little, because I've contributed once to it. I don't want to belittle your project, but I'd like to know: what are the differences between gitgo and gogit ?
(I see at least one similarity: the name is extraordinarily unimaginative :)
There are issues. And there is the legendary "pack v4" (current pack versions are 2 or 3) but it still a work in progress.
So in the Go ecosystem it is actually preferred to use pure Go libraries not to loose these benefits.
If the internal documentation include any creative elements which are reflected by the git implementation, then the clean-room would be contaminated.
More detail than that would require reading the source code..