News for hackers. Stuff that matters.
Re:Git’s database internals II: commit history queries by ByronBates on Tuesday August 30, 2022 @09:37PM (#32657355)
> This is fast enough for small repositories, but as the repository increases in size the overhead of parsing these plain-text files to get the graph relationships becomes too expensive. Even the fact that we need a binary search to locate the object within the packfile begins to add up.

From my experience and measurements as the author of `gitoxide` both the parsing of commits as well as the binary search to find the object are negligible costs here. The majority of the time is spent in `zlib` which has to inflate objects and deltas prior to use. This makes me wonder what `zstd` would do to these kinds of workloads.

In any case, the object database decoding performance is the bottleneck when traversing commits, and one can expect to get about 120000 commits per second on the linux kernel pack on a modern CPU core like M1 with `gitoxide`s implementation.

Besides that, I find the explanation of the commit-graph file interesting and how it relates to commit-graph queries. There is so much to learn, and so much still to implement :).

Re:Git’s database internals II: commit history queries by dralley on Tuesday August 30, 2022 @11:09PM (#32657938)
I wonder if zlib-ng would make a difference, since it has a lot of optimizations for modern hardware.

https://github.com/zlib-ng/zlib-ng/discussions/871

Re:Git’s database internals II: commit history queries by ByronBates on Wednesday August 31, 2022 @12:41AM (#32658448)
That number is already based on using `zlib-ng`. Sometimes I wonder if it's cheating since `git` might not actually use it. In any case, improving the efficiency of `zlib` along with the Rust integration for it has immediate impact on the git object database performance of `gitoxide`.
Re:Git’s database internals II: commit history queries by svnpenn on Tuesday August 30, 2022 @11:35PM (#32658085)
gitoxide is a cool project, but it cant even clone yet:

https://github.com/Byron/gitoxide/discussions/300

Re:Git’s database internals II: commit history queries by ByronBates on Wednesday August 31, 2022 @12:46AM (#32658472)
`gitoxide` can currently perform many of the tasks required, and closing the loop for a full clone with worktree is going to happen at the end of this year.

That said, here is how to receive a pack from the remote and resolve it: `mkdir out && gix -v no-repo pack receive https://github.com/Byron/gitoxide out`.