News for hackers. Stuff that matters.

A comment on We Put Half a Million Files in One Git Repository, Here's What We Learned (2022). Parent comment

Re:We Put Half a Million Files in One Git Repository, Here's... by PMunch on Monday August 28, 2023 @07:42AM (#37292563)
Wouldn't Git LFS be the tool for this job? Have the automated tool build a .zip file for example of the translations (possibly with compression level set to 0), then have your build toolchain unzip the archive before it runs. Then check that big .zip file into GitLFS, et voila you now have this large file versioned in Git.
[ Parent | on HN ]
Re:We Put Half a Million Files in One Git Repository, Here's... by folmar on Monday August 28, 2023 @08:42AM (#37293098)
It's good enough for the small usecases, but way behind tools that have first class support for binary files (binary deltas, common compression, ...). Even SVN shines here.
[ Parent | on HN ]
Re:We Put Half a Million Files in One Git Repository, Here's... by maccard on Monday August 28, 2023 @08:01AM (#37292732)
Git LFS isn't the same as git, though. It's better than putting everything in a separate store, but for one it disables offline work, and breaks the concept of D in the DVCS of git.

> then have your build toolchain unzip the archive before it runs

My build toolchain shouldn't have to work around the shortcomings of my environment, IMO.

> et voila you now have this large file versioned in Git.

No, it's on a separate http server that is fetched via git lfs. Subtle, but important difference.

[ Parent | on HN ]
Re:We Put Half a Million Files in One Git Repository, Here's... by aidenn0 on Monday August 28, 2023 @01:00PM (#37297181)
> it disables offline work,

This is a non-issue for images and autogenerated files, since you shouldn't ever be doing a merge on them.

> breaks the concept of D in the DVCS of git.

git-annex is distributed and works well for files that will never be merged (such as images, or autogenerated files)

[ Parent | on HN ]
Re:We Put Half a Million Files in One Git Repository, Here's... by haizzz on Wednesday August 30, 2023 @01:34AM (#37318078)
Separately we also found that git lfs is not very optimised for large repositories, notably its locking feature which list every file tracked by git for every checkout and commit command.
[ Parent | on HN ]