Decreasing Gitlab repo backup times from 48 hours to 41 minutes

Re:Decreasing Gitlab repo backup times from 48 hours to 41 m... by 6LLvveMx2koXfwn on Friday June 06, 2025 @11:22PM (#44207162)

They weren't, if you look at the fix [1] the dedupe loop was run in all cases, not just those with known dupes, so the performance hit was any bundle with lots of refs.

1.https://github.com/git/git/commit/bb74c0abbc31da35be52999569...

[ Parent | on HN ]

Re:Decreasing Gitlab repo backup times from 48 hours to 41 m... by rjmunro on Saturday June 07, 2025 @02:21PM (#44211490)

But why couldn't they just dedupe the refs from the command line before starting the actual bundling - surely there are never more than a couple of hundred of those (one per branch)?

[ Parent | on HN ]

Re:Decreasing Gitlab repo backup times from 48 hours to 41 m... by 6LLvveMx2koXfwn on Sunday June 08, 2025 @10:14AM (#44217157)

The point is the performance hit had nothing to do with dupe count (which could be zero), and everything to do with ref count.

[ Parent | on HN ]

Re:Decreasing Gitlab repo backup times from 48 hours to 41 m... by nayak on Monday June 09, 2025 @04:28AM (#44222445)

Spot on. Some of our repositories at GitLab can contain millions of references.

[ Parent | on HN ]