News for hackers. Stuff that matters.
Re:Supercharging the Git Commit Graph IV: Bloom Filters by lozenge on Tuesday July 17, 2018 @03:29AM (#17547963)
Does it work when I'm looking for all commits changing a directory? I.e. Do directories get added to the bloom filter or just filenames?
Re:Supercharging the Git Commit Graph IV: Bloom Filters by stolee on Tuesday July 17, 2018 @06:33AM (#17548644)
It does! The Bloom filter stores every path that changed, including paths to trees (except the root tree, which is expected to be changed by default). When you change a file, you also change every tree above it.
Re:Supercharging the Git Commit Graph IV: Bloom Filters by kingosticks on Tuesday July 17, 2018 @03:40PM (#17552550)
So presumably that eats into the 512 item limit. If I had quite a deep directory structure (10 levels doesn't sound that unreasonable) I'd only get this speedup for commits changing 51 files or less. Right?
Re:Supercharging the Git Commit Graph IV: Bloom Filters by stolee on Wednesday July 18, 2018 @06:33AM (#17557004)
In your calculation, you are assuming that all of those files are in independent parts of the folder hierarchy. In practice, deep directory structures mean that single commits are likely focused in a few very deep folders. We ran the statistics on a number of repos with different shapes to come up with this number, and it covers 99% of commits that we measured.
Re:Supercharging the Git Commit Graph IV: Bloom Filters by kingosticks on Wednesday July 18, 2018 @08:46AM (#17557834)
That is true and I do agree. Thanks for clarifying and for the informative write-up, it's a great use of bloom filters.
Re:Supercharging the Git Commit Graph IV: Bloom Filters by tempay on Tuesday July 17, 2018 @04:15AM (#17548135)
I thought git didn't have a concept of directories, just blobs that have a path? If so, I presume "all commits changing a directory" is found by individually checking each file contained by the directory.
Re:Supercharging the Git Commit Graph IV: Bloom Filters by masklinn on Tuesday July 17, 2018 @05:23AM (#17548399)
Git very much has directories, it calls them "tree": a commit links to a tree, which links to a number of trees or blobs associating each with a name: https://git-scm.com/book/gr/v2/Git-Internals-Git-Objects

A blob is a file (ish, it can also be use for some other things since it's not intrinsically named), a tree is a directory.

However Git doesn't treat filesystem directories themselves as first-class e.g. you can't version an empty directory. A tree only exists in order to contain sub-items (ultimately blobs).

Re:Supercharging the Git Commit Graph IV: Bloom Filters by lozenge on Tuesday July 17, 2018 @04:45AM (#17548256)
As the article mentions, each folder is a tree object containing (referencing) subfolders and blobs (files). Run git show HEAD^{tree} to see it.

Finding the list of all filenames that were ever in a particular path is very expensive.