Important point here is that if:

- you have lots of large files which are not amenable to diff and change frequently;

- everyone is working within the same company, and usually on the same network;

then a DVCS is unhelpful because you have to spend the disk cost for the repositories on every machine having a full copy of everything that's ever been checked in, regardless of whether they need it or not.

Many games are tens of gigabytes when shipped. It's easy to imagine a process which accumulated hundreds of megabytes of asset changes every single day over a multi-year development process. Then you can imagine having to have expensive terabyte SSDs just to work on it with all your tools.

I'm actually looking at this problem at work for possibly converting a large repository from svn which is a decade old and merely tens of gigabytes. Frankly svn handles it just fine so I'm going to defer the problem until I absolutely have to migrate.

As a point of reference, here are some sizes for the repos of the game I work on which has been developed over around 11 years.

120 GB for the game data repo. This is the repo for the source and art that actually ships.

2.5 TB for the raw art repo. This is where the pre-export raw files go.

Obviously it would be impractical to require that the artists all have the entire repo. We use SVN.

> which has been developed over around 11 years.

are there that many games which have a lifetime of over 11 years? I only know about one...

There are many. Online games have very long lifetimes.

Think of League of Legends, Team Fortress 2, EVE online, World of Warcraft. There are hundreds more smaller games.

well lifetime = time where stuff is still under development. LoL = not more than 11 years old (heck it's not even 10 years old) TF2 had it's 10th anniversary this year. which means eve online, WoW and runescape still get patches/updates/etc.

as said there aren't that many games that have a lifespan of over 11 years.

You've been given plenty of examples showing it's been done. What's the point you're trying to reach?

League of Legends started development in 2006 and TF2 started development in 1998 or 2005 depending on if you start counting from the redesign or not.

There are so many games that have this long a lifespan.

I still play CS 1.6 which is getting close to 20 years old.

cs 1.6 is not under active development.

yearly sports games like baseball mogul change yearly but same base

Someone else’s mentionec the size of their repo on a game that’s 11 years old. Last time I checked on an unreleased project, in development for about 4 years at this stage, we have 200GB of source code and binaries checked into perforce for normal usage, and about 800GB of source assets (audio, models, textures). We probably 150 changes a day, and about half of them are to art content, the other half to code. There’s a lot of churn, and this is on a “small” AAA project (probably about 100 people total).

I actually setup a small studio to work exclusively on a Mercurial largefiles-based VCS. I would say its one of the best solutions today on the market for self-hosted free users.

Why? Perforce's integration to Unity is quite poor (they have a huge untapped market here), so you end up having to resolve a lot of things slowly in their tools. Git/HG are much faster in my experience at detecting changes and interacting sanely with them. Also the team could never learn why a file is checked out, why can't they commit, etc.

We regularly clean out our largefiles cache on disk, so most of the time everyone just has the latest version of a given binary file on disk. The server of course has every revision, but I want that.

And most important of all: with small tweaks we're able to use Phabricator for all of our task management/documentation workflow. Getting VCS hooks out of the box to let artists say "Adding typewriter model, please review T555" in their commit and having that task automatically get assigned to the reviewer is priceless.

Most of my team doesn't have any idea what VCS is, but they've learned to use TortoiseHg (they call it "the turtle") and Phabricator to organize ourselves.

While Mercurial isn't the only way to get there, its free, its fast, its simple, and it unlocks the power of Phabricator (so does Git+LFS I believe).

So in my experience, I would say hg+largefiles is an excellent solution for game development.

The only free option with unlimited storage seems to be Microsoft TeamServices with Git-LFS support. (1) (2)

I have been using Bitbucket and Mercurial for all side projects of mine for quite a while. But when you start with game development, you will reach the repository limits quite fast. Textures, meshes, sound, music, concept art and other binary blobs eat alot of storage.

Git-LFS is a bit a pain to setup, because you need to define before checking in, which extensions need to be stored as large file. And then there are check-in hooks, which sometimes did seem unreliable. Visual Studio git integration is also quite limited, but SourceTree did serve me well.

It's quite liberating if you're able to check in code and assets together without taking into account the space needed.

1) https://blogs.msdn.microsoft.com/devops/2015/10/01/announcin...

2) https://www.visualstudio.com/vso/

BitKeeper solved this years ago with one more centralized servers that hold the binaries. When you clone you only get the tip but you can retrieve any version you want when you need it. Scales to terabytes easily.

Free and open source (apache v2) at http://bitkeeper.org

Blizzard gave a great GDC talk titled “The Data Building Pipeline of 'Overwatch'”. It covered their in-house, http-based asset distribution system.

Take-aways: http://seanmiddleditch.com/my-gdc-17-talk-retrospective/

https://twvideo01.ubm-us.net/o1/vault/gdc2017/Presentations/...

Pretty interesting.

Game Development seems to have such a different workflow than most of the stuff I'm familiar with like backend, web dev, and the occasional network programming.

https://gamedev.stackexchange.com/questions/480/version-cont... This link recommends peforce as the standard .

What do people in the industry actually use ?

Perforce, PlasticSCM, SVN or git + SVN. That probably covers 95%.

The small dev I work for uses SVN, though they tried standard git for another project which by all accounts quickly ran into the size issues with the repository.

For 2D graphics, SVG (and more generally vector graphics) fit very well in git. For 3D, there are models, but there's still the problem of textures.

Is there anything close to a text-based procedural texture format ? Textures could be procedurally generated at startup and transformed into bitmaps. I am aware of kkrieger, but is there anything other than proof of concept ? No one takes voxels seriously anymore...

I’ve been checking in gigabyte large assets into my git repos with Git-LFS and self-hosted GitLab, and it’s been working fine for now.

Are there any issues with this approach I should be aware of, considering that Hg with Largefiles seems to have some, too?

Not explicitly, IFAIK the main slowness/limits you risk hitting are:

1. Git clones are deep and wide by default, so you end up with local repos that are the size of the total history. (easy to work around)

2. Git narrow clone support isn't the greatest, so you will probably want to fit the "working directory" on one computer. (somewhat painful to work around)

3. Git has to checkout all files every time you change branches which can be slow if you have a large working directory. (very painful)

4. Many operations (status, diff, commit for example) require git to scan the working directory to see what changed. This will be very slow if you have many files in your working directory. (very painful)

Luckily all of these problems can be solved seamlessly with a virtual filesystem approach. For example https://github.com/Microsoft/GVFS

> Okay, so once we have started, the most important thing to know is: This does not in any way change how Hg handles files in memory. [...] Hg has to take the file and consume many times more memory during the commit then the size of the file, to try to figure out what the differences are.

This isn't true. Largefiles aren't stored as deltas, but as complete blobs. Mercurial still reads them in their entirety, so it still uses a lot of memory, but it's not diffing.

> The next problem is everyone collaborating on the project would have to take a huge Pull with the new large files, for every version of the large file they don't yet have [...] if you want to go back to a revision you haven’t pulled yet and the Server is not up you’re out of luck. That means you should get all the commits at some point anyway (because you want all the code versions at your side), so what’s the point?

Largefiles's mechanism doesn't require that you download every version of every large file. That's a key part of its design. If you decide that as a matter of policy that you want to download them all anyway, then no, largefiles won't help much.

> Well, they handle files by placing them outside the repo, and storing only the hash of the file in the repo itself (all bigfile hashes inside one file). This has an unfortunate effect that you won't be able to tell which exact bigfile/largefile has actually been modified when looking in the history – the only thing you'd see in the repo is the cumulative file that holds the hashes of all bigfiles as having a change.

This isn't true either. Largefiles stores the hashes in separate files, and the history machinery is able to interpret the records properly.

I've written a little script to demonstrate the structure of a largefiles repository:

https://bitbucket.org/snippets/twic/7eeAxy

One thing that would be really useful that largefiles doesn't (that i know of) do would be to opt out of downloading some largefiles at all. If i'm checking out an old revision just to read some old code, i don't want to spend ages pulling largefiles that i'm not going to look at. You can do this with Facebook's remotefilelog extension, which lets you make shallow clones, which can omit the large files, but it's awkward:

https://bitbucket.org/facebook/hg-experimental/src/default/r...

Would a maven like repository possibly be a solution to this problem? Simply have dependencies to binary files/artifacts distributed by Nexus?

File system Snapshots (like in ZFS) would work nice with binary data.