Re:Mercurial with Largefiles: Why it is not a solution for g...
by twic on Sunday December 03, 2017 @06:14PM
(#15840216)
> Okay, so once we have started, the most important thing to know is: This does not in any way change how Hg handles files in memory. [...] Hg has to take the file and consume many times more memory during the commit then the size of the file, to try to figure out what the differences are.
This isn't true. Largefiles aren't stored as deltas, but as complete blobs. Mercurial still reads them in their entirety, so it still uses a lot of memory, but it's not diffing.
> The next problem is everyone collaborating on the project would have to take a huge Pull with the new large files, for every version of the large file they don't yet have [...] if you want to go back to a revision you haven’t pulled yet and the Server is not up you’re out of luck. That means you should get all the commits at some point anyway (because you want all the code versions at your side), so what’s the point?
Largefiles's mechanism doesn't require that you download every version of every large file. That's a key part of its design. If you decide that as a matter of policy that you want to download them all anyway, then no, largefiles won't help much.
> Well, they handle files by placing them outside the repo, and storing only the hash of the file in the repo itself (all bigfile hashes inside one file). This has an unfortunate effect that you won't be able to tell which exact bigfile/largefile has actually been modified when looking in the history – the only thing you'd see in the repo is the cumulative file that holds the hashes of all bigfiles as having a change.
This isn't true either. Largefiles stores the hashes in separate files, and the history machinery is able to interpret the records properly.
I've written a little script to demonstrate the structure of a largefiles repository:
One thing that would be really useful that largefiles doesn't (that i know of) do would be to opt out of downloading some largefiles at all. If i'm checking out an old revision just to read some old code, i don't want to spend ages pulling largefiles that i'm not going to look at. You can do this with Facebook's remotefilelog extension, which lets you make shallow clones, which can omit the large files, but it's awkward:
This isn't true. Largefiles aren't stored as deltas, but as complete blobs. Mercurial still reads them in their entirety, so it still uses a lot of memory, but it's not diffing.
> The next problem is everyone collaborating on the project would have to take a huge Pull with the new large files, for every version of the large file they don't yet have [...] if you want to go back to a revision you haven’t pulled yet and the Server is not up you’re out of luck. That means you should get all the commits at some point anyway (because you want all the code versions at your side), so what’s the point?
Largefiles's mechanism doesn't require that you download every version of every large file. That's a key part of its design. If you decide that as a matter of policy that you want to download them all anyway, then no, largefiles won't help much.
> Well, they handle files by placing them outside the repo, and storing only the hash of the file in the repo itself (all bigfile hashes inside one file). This has an unfortunate effect that you won't be able to tell which exact bigfile/largefile has actually been modified when looking in the history – the only thing you'd see in the repo is the cumulative file that holds the hashes of all bigfiles as having a change.
This isn't true either. Largefiles stores the hashes in separate files, and the history machinery is able to interpret the records properly.
I've written a little script to demonstrate the structure of a largefiles repository:
https://bitbucket.org/snippets/twic/7eeAxy
One thing that would be really useful that largefiles doesn't (that i know of) do would be to opt out of downloading some largefiles at all. If i'm checking out an old revision just to read some old code, i don't want to spend ages pulling largefiles that i'm not going to look at. You can do this with Facebook's remotefilelog extension, which lets you make shallow clones, which can omit the large files, but it's awkward:
https://bitbucket.org/facebook/hg-experimental/src/default/r...