News for hackers. Stuff that matters.
Re:Scaling Mercurial at Facebook by Crito on Tuesday January 07, 2014 @05:18PM (#7020395)
That is certain. If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go.

The problem is "how do you grow even further?" There isn't a perfect solution to that (yet), but I think the only existing workable (although absolutely imperfect) solution is splitting your codebase into many small repos.

Thankfully very few companies need to worry about this problem right now. A single Mercurial repo is apparently scaling for Facebook currently, and few companies will manage to stress Perforce.

Re:Scaling Mercurial at Facebook by moron4hire on Wednesday January 08, 2014 @09:38AM (#7023668)
At what point do we say "enough" though? I mean, start from the absolute maximum: how would one make a system to control all of the source code in the world?

For all of the source code in the world that is source-controlled, it exists in separate repos. It doesn't exist in one repo, and where there are inter-project dependencies, it is the dependency consumer's responsibility to keep abreast of the changes in the dependency and integrate them as necessary. It is self-organizing.

So, if Google and Facebook have grown beyond what can be done with a single repo, either they are not being mindful of their engineering practices (which puts the public at significant risk), or they are approaching a scale that is more similar to "world scale" than it is "corporate scale".

Which I would also say is putting the public at significant risk.

Re:Scaling Mercurial at Facebook by Crito on Wednesday January 08, 2014 @12:58PM (#7025156)
This is a very valid point. It is not clear at all that a single repo for the world is desirable in the long run, despite some of the individual advantages that sort of setup has. It is part of the reason that I don't think we will see the "next generation" of VCS anytime soon. (Which in turn makes me think that "holding out" for those systems, hoping they will rescue you from your scaling problem, is a bad decision. It is best to move to multiple repos sooner rather than later).
Re:Scaling Mercurial at Facebook by VladRussian2 on Tuesday January 07, 2014 @05:34PM (#7020482)
> If you are trying to run at a very large scale with a single repo, Perforce (or apparently Mercurial) is the way to go.

that is the problem - single logical repo in Perforce/Mercurial does mean single physical repo. Which obviously causes issues at very large scale. Even at normal enterprise scale :)

Git solves it through kind of distributed scaling where many operations can be performed on local physical repos without ever hitting the central. With centralized solutions, like Perforce, Mercurial, etc... most of the operations are performed on central server and your ability to scale vertically hits the ceiling pretty soon.

Compare the simplest case - each dev updating his local workspace/repo would hit the central server in Perforce where is with Git you can (and normally would have) a small set of downstream repos which the updates would be propagated/distributed through. You can branch left and right in Git in your local and team's repos without "master" repo and "central" server involved (while all your branch activity in Perforce is happening right on the central server in the central repo). The same happens for synchronizing your work inside the team - no need to hit the central. Etc...

Re:Scaling Mercurial at Facebook by ngoldbaum on Tuesday January 07, 2014 @06:00PM (#7020610)
Mercurial is a DVCS, just like git. It has no concept of a central server.
Re:Scaling Mercurial at Facebook by VladRussian2 on Tuesday January 07, 2014 @06:17PM (#7020698)
>Mercurial is a DVCS

my bad. 5+ years since i worked with Mercurial. Digging deep into painful memories, my Mercurial PTSD from that time is absence of in-repo branching - need to clone which is a killer for very large repo we had and no partial commits - again a killer aggravated by the above mentioned absence of in-repo branches. Both issues made working with large repo unreasonably and unnecessary hard.

Re:Scaling Mercurial at Facebook by sampo on Tuesday January 07, 2014 @06:55PM (#7020847)
I am sure Mercurial has always had (named) in-repo branches, so you were using Mercurial in strange manner.
Re:Scaling Mercurial at Facebook by twic on Tuesday January 07, 2014 @07:20PM (#7020950)
There was a time when the developers advised using parallel clones rather than named branches, though. I never really understood why.

These days, the standard practice is probably to use bookmarks, which are more or less like Git branches.

Re:Scaling Mercurial at Facebook by rbehrends on Tuesday January 07, 2014 @08:51PM (#7021206)
These days, the standard practice is probably to use bookmarks, which are more or less like Git branches.

No. You have the choice to use either named branches or bookmarks, and which one you choose is a matter of your workflow. Note that even a Git-like workflow does not necessarily require the use of bookmarks.

Mercurial is still designed under the assumption that people will use named branches and the Git-like aspects of bookmarks exist to ease transition from Git: http://www.selenic.com/pipermail/mercurial/2014-January/0464...

Personally, I use Fossil-like private branches [1] in order to facilitate local work that I may still want to reorganize before pushing it to a shared server; I use bookmarks solely for local tags.

[1] Emulated by putting a named branch in the secret phase. I actually have a simple extension to automatically make all branches that start with a dot (e.g, ".tempstuff") secret.

Re:Scaling Mercurial at Facebook by pnathan on Tuesday January 07, 2014 @06:40PM (#7020797)
hg certainly has had branches since 2008 at least.

What do you mean by partial commit? I can guess but I would rather know. :-)

Re:Scaling Mercurial at Facebook by charlesma on Tuesday January 07, 2014 @06:07PM (#7020646)
> With centralized solutions, like Perforce, Mercurial, etc...

I think you're incorrectly putting mercurial and perforce in the same bucket. Mercurial is a lot more like Git than perforce in that it's a DVCS. Facebook made it scale by using a lot of centralized caching, but most operations (diff, looking through history, branching, etc) can be done locally.

Re:Scaling Mercurial at Facebook by Sssnake on Tuesday January 07, 2014 @06:31PM (#7020761)
Git doesn't solve the issue, it suffers from it, that's the point. And as others pointed out already, mercurial is a DVCS. It just isn't as shitty as git, that's all.
Re:Scaling Mercurial at Facebook by cstejerean on Tuesday January 07, 2014 @06:46PM (#7020819)
Mercurial itself runs into issues way before git does, it's just that Facebook has essentially gutted a bunch of things out Mercurial in order to make the resulting thing fast (punting on things like computing status, downloading diffs, etc).
Re:Scaling Mercurial at Facebook by Sssnake on Tuesday January 07, 2014 @08:20PM (#7021111)
No, git runs into issues before mercurial does. Just saying the opposite of reality doesn't make it so.
Re:Scaling Mercurial at Facebook by ibrahima on Wednesday January 08, 2014 @12:37AM (#7021903)
Did you even read the post? Stock git is significantly faster than stock mercurial, it's only after they hack it up that they make it faster.
Re:Scaling Mercurial at Facebook by jordigh on Wednesday January 08, 2014 @08:39AM (#7023347)
There are some operations in which git is way slower than Mercurial. Two examples are cloning and blame/annotate. git's data structures are optimised for some operations, but not for others.
Re:Scaling Mercurial at Facebook by Sssnake on Wednesday January 08, 2014 @06:55PM (#7027764)
Did you? It isn't about speed, it is about scaling.