News for hackers. Stuff that matters.
Re:Supercharging the Git Commit Graph IV: Bloom Filters by avar on Tuesday July 17, 2018 @05:18AM (#17548377)
[Source: I contribute to Git and talked to Twitter's "git guys" at the time at Git Merge, am not affiliated with Twitter in any way]

My understanding is that they didn't cargo-cult anything. They made a conscious choice at Twitter to pursue the monorepo model.

One reason they mentioned is that while they have a lot of services running all over the place, those services tend to heavily use the same underlying base libraries.

Think the library that validates Twitter usernames, parses out the text of a tweet etc. By having a monorepo they can easily atomically change those libraries for all their consumers. This tends to be why companies go for the monorepo model in general.

This is all information from 2016-ish. Around that time users with *@twopensource.com E-Mail addresses stopped contributing to git.git. I have no idea why, presumably changed internal priorities or something like that. Maybe their internal repository structure changed so they didn't need to author their own performance patches anymore, maybe not...

Re:Supercharging the Git Commit Graph IV: Bloom Filters by influx on Tuesday July 17, 2018 @09:41AM (#17549641)
Sure, that was the justification, but when you literally can’t switch branches without waiting for a minute, I fail to see the benefit.

As I recall the developers authoring the custom git patches left the company around that time. I don’t know what happened after that. There was talk of moving to mercurial with facebooks patches, but I had already left.

Re:Supercharging the Git Commit Graph IV: Bloom Filters by avar on Wednesday July 18, 2018 @08:40AM (#17557783)
It would be interesting to get some background info on that branch switching.

Switching branches per-se is really cheap in git, on linux.git it takes 200ms, around the time it takes to run a cold "git status". This is because just creating a new branch doesn't need to touch the tree at all.

I know "status" was a bottleneck at Twitter. They had the first inotify patches to git, but it never made it in. Eventually the patches Microsoft wrote to do the same thing made it in.

What can get expensive is if the tree you're switching to has drastically different content. On the latest linux.git (~60k files) switching to the 2.6.* era takes around 10 seconds or me (~10k files).

I'd expect with a monorepo model like what Twitter had (has?) that most working branches are relatively up-to-date with the master branch, so switching should be cheap.

Was it spending most of its time in the "Checking out files" phase, or before that?

Was this on e.g. OSX with some corporate virus scanner running where each I/O syscall was wrapped? That can drastically slow things down.

Or was it just that the repository truly had a ridiculous amount of files in the checkout (around 1 million?).

Edit: I remember now that I have an old copy of 2015-04-03-1M-git.git which David Turner of Twitter publicly shared a while back, it was meant to emulate the size and shape of Twitter's monorepo. It has around 230k files.

Re:Supercharging the Git Commit Graph IV: Bloom Filters by stochastic_monk on Tuesday July 17, 2018 @11:57AM (#17550573)
Why didn’t they use submodules, then? It doesn’t force the code into the same git history but makes shared code easily deployable.