We Put Half a Million Files in One Git Repository, Here's What We Learned (2022)

Re:We Put Half a Million Files in One Git Repository, Here's... by Karellen on Monday August 28, 2023 @09:16AM (#37293449)

> In my last org, we vendored our entire toolchain,

You vendored all your compilers/language runtimes in the source control repo of each project? Including, like, gcc or clang? WTF?

> It's a shame that the mantra of "do one thing well" devolves into "only support a few hundred text files on linux" with git.

Because the Linux kernel source tree and its history can accurately be described as "a few hundred text files".

Yeah, right.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here's... by tom_ on Monday August 28, 2023 @09:33AM (#37293691)

I've worked on a couple of game projects that did this. Build on Windows PC, build for Windows/Switch/Xbox One/Xbox Serieses/PS4/PS5/Linux. I was never responsible for setting this up, and that side of things did sounda bit annoying, but it seemed to work well enough once up and running. No need to worry about which precise version of Visual Studio 2019 you have, or whether you've got the exact same minor revision of the SDK as everybody else. You always build with exactly the right toolchain and SDK for each target platform.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here's... by 0xcoffee on Monday August 28, 2023 @09:24AM (#37293573)

It's not that unusual, we vendor entire VM images which contain the development environment. (Codebase existed since before docker). And it works well, need to fix something in a project that was last update 20 years ago? Just boot up the VM and you are ready.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here's... by xorcist on Monday August 28, 2023 @11:05AM (#37295242)

I don't think that was the question but rather why commit to git?

Having local commits intermingled with an upstream code base can make for really hairy upgrades, but I guess every situation is slightly different.

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here's... by maccard on Monday August 28, 2023 @11:18AM (#37295500)

> but rather why commit to git?

Well we don't put them in git, we put them in perforce because git keels over if you try and stuff 10GB of binaries into it once every few months.

I think the real question is the other way around though, why _not_ use git for versioning when that's what it's supposed to be for? Why do I have to verison some things with git, and others with npm/go build/pip/vcpkg/cargo/whatever?

[ Parent | on HN ]

Re:We Put Half a Million Files in One Git Repository, Here's... by maccard on Monday August 28, 2023 @11:11AM (#37295355)

> You vendored all your compilers/language runtimes in the source control repo of each project? Including, like, gcc or clang? WTF?

Yep. Along with paltform SDKs, third party dependencies, precompiled binaries, non-redistributable runtimes, you name it.

Giant PSD or FBX files? 4K Textures? all of it.

Client mappings are the bread and butter of P4 (or Stream views more recently which are not as nice to work with) - you say "I don't want the path containing MacOS" if you don't want it.

> Because the Linux kernel source tree and its history can accurately be described as "a few hundred text files".

I was off by a little bit, it's ~60k. But it's still "only" 60k text files, no matter how important those text files are.

[ Parent | on HN ]