News for hackers. Stuff that matters.

A comment on The Future of Large Files in Git Is Git. Parent comment

Re:The Future of Large Files in Git Is Git by tempay on Friday August 15, 2025 @06:55PM (#44918223)
> binary chunking and deduplication

Are there many binaries that people would store in git where this would actually help? I assume most files end up with compression or some other form of randomization between revisions making deduplication futile.

[ Parent | on HN ]
Re:The Future of Large Files in Git Is Git by digikata on Friday August 15, 2025 @07:18PM (#44918380)
I don't know, it's all probability in the dataset that makes one optimization strategy better over another. Git annex iirc does file level dedupe. That would take care of most of the problem if you're storing binaries that are compressed or encrypted. It's a lot of work to go beyond that, and probably one reason no one has bothered with git yet. But borg and restic both do chunked dedupe I think.
[ Parent | on HN ]
Re:The Future of Large Files in Git Is Git by hinkley on Friday August 15, 2025 @07:11PM (#44918327)
It would likely require more tooling.
[ Parent | on HN ]
Re:The Future of Large Files in Git Is Git by adastra22 on Friday August 15, 2025 @08:29PM (#44918816)
A lot in the game and visual art industries.
[ Parent | on HN ]
Re:The Future of Large Files in Git Is Git by zigzag312 on Saturday August 16, 2025 @05:42AM (#44921786)
2-3x reduction in repository size compared to Git LFS in this test:

https://xethub.com/blog/benchmarking-the-modern-development-...

[ Parent | on HN ]