News for hackers. Stuff that matters.
Re:Scaling Git’s garbage collection by 1letterunixname on Tuesday September 13, 2022 @11:16PM (#32832934)
Most people should `git clone --depth 1` most of the time. Large businesses should do trunk-based development with something other than git that scales using distributed file systems with phantom checkouts, negating the need for LFS blob stores.
Re:Scaling Git’s garbage collection by iudqnolq on Wednesday September 14, 2022 @08:05AM (#32835992)
I haven't been able to find a link recently, but I recall from a few years ago that that's much harder on GitHub's servers. One of the newer package managers that uses GitHub for hosting (brew maybe?) tried to be helpful and default all their users to shallow clones. GitHub people wanted them to push an update ASAP to go back to full clones, and there was an issue thread laying it all out.

If I recall correctly it has to do with the way they cache git. If you request shallow clones they have to run a git process each time to make a custom download with only the data you need. Full clones get served from a cache.

Re:Scaling Git’s garbage collection by masklinn on Wednesday September 14, 2022 @01:34PM (#32840493)
It was an issue with Cocoapods: https://github.com/CocoaPods/CocoaPods/issues/4989#issuecomm...

> If I recall correctly it has to do with the way they cache git.

It was not actually the shallow clone, the shallow clone is fine.

The problem was shallow fetches afterwards, as it made computing the minimum set of changes to fetch much harder for git (during the fetch operation the client and server actually try to negotiate a minimum set of changes, and the server creates an ad-hoc pack for this).

Not only that, but they'd hit a git edge case which ended up causing disproportionate CPU usage and converting the shallow clones to near-full clones but very, very inefficiently.

This issue was compounded by the very git-unfriendly layout of the repository: one of the directories had >16000 subdirectories, something Git's tree-processing code apparently was not much tested for, leading to significant inefficiencies.

Re:Scaling Git’s garbage collection by vildravn on Wednesday September 14, 2022 @03:15AM (#32834190)
> with something other than git that scales using distributed file systems with phantom checkouts, negating the need for LFS blob stores.

What would that something be for example?

Re:Scaling Git’s garbage collection by 1letterunixname on Wednesday September 14, 2022 @03:21AM (#32834228)
NDA, so I can't talk about it. I would like to but I can't.
Re:Scaling Git’s garbage collection by saagarjha on Wednesday September 14, 2022 @06:15AM (#32835191)
Not that you should do anything you're not comfortable with but I will note that the source control systems for most major companies are essentially known, especially around here. "At Facebook we use a Mercurial-inspired thing with some local extensions" is not really earth-shattering.
Re:Scaling Git’s garbage collection by tex0 on Wednesday September 14, 2022 @04:53AM (#32834713)