News for hackers. Stuff that matters.
Re:Mercurial 4.0 Sprint Notes by agentgt on Tuesday October 18, 2016 @01:05PM (#12736514)
I know this is slightly tangental but I'm always a little shocked that Facebook (and I think Google to some extent) have massive mono repositories.

The benefits of having one repository do not seem to be worth the serious performance issues as well as potential coupling that can happen with a gigantic code base as well also making much more difficult to OSS certain parts.

e.g. why doesn't FB use dependency management (or binary package management aka maven, npm, etc)? That is have multiple repositories and cut releases. Build tools to help cut and manage the releases and dependency graph of the releases.

There are even plugins and shell scripts that will make Mercurial act like a mono repository for many small repositories (I use it for our own Java Maven code base).

I must be missing some killer features and would love to see it in action (FB major repository).

Re:Mercurial 4.0 Sprint Notes by jcranmer on Tuesday October 18, 2016 @01:18PM (#12736661)
Anyone who is old enough to remember the pain of VCSes like CVS and RCS will note that the number one feature touted to move to any other VCS is invariably "atomic changesets"--all the necessary changes to files are listed in a single changeset. Monorepos are nothing less than remembering the value of those atomic changesets.

An example of utility of monorepos is things like automation--if you change, for example, how you publish packages, and you need to maintain multiple stable branches, it is immensely useful to keep the automation steps in the same repository as code. If you don't, your automation repository then looks like

    if version < 31:
        step_a()
        step_b()
        step_c()
    elif version < 35:
        step_a()
        step_bv2()
        step_d()
    else:
        ...
which quickly grows unmaintainable at scale.
Re:Mercurial 4.0 Sprint Notes by brazzledazzle on Tuesday October 18, 2016 @10:13PM (#12740473)
I can only imagine that the only way to scale that would to make whoever owns the repo responsible for automation as well. A dedicated team can provide the tooling/frameworks (basically a "golden path") but if/how would need to be up to the team. I think in a general sense this is the approach Netflix takes to services.

That said I have no idea how Netflix does source control, only that they use BitBucket/Stash on-prem and that supports both mercurial and git.

Re:Mercurial 4.0 Sprint Notes by agentgt on Wednesday October 19, 2016 @08:17AM (#12743080)
If you do it right your automation steps are just another package dependency that is versioned (I know because this is how we do it). We have a single bash script that will auto update itself (think akin to home brew albeit in bash).

The only immensely dangerous thing that can happen is if you drop your package repository or change formats of the repository which rarely happens.

Re:Mercurial 4.0 Sprint Notes by the_duke on Tuesday October 18, 2016 @01:40PM (#12736872)
When you split your libraries into separate projects, you have to start versioning them.

And every update to the code requires other teams to then update their library/app to use the new version, and apps depending on that dependency... you get my drift.

You end up with a complicated dependency headache which hurts productivity.

With a monorepo, you can statically identify all places where your library is used and update those automatically with tooling, or manually. You can also monitor where and how a piece of code is used across the company, etc.

If you then ensure that only commits get accepted that pass the relevant tests, you end up with a sane and working HEAD that's always up to date.

Of course there are lot's of drawbacks as well.

But Google and FB seem to have concluded that this is the better approach for them.

Re:Mercurial 4.0 Sprint Notes by agentgt on Wednesday October 19, 2016 @08:35AM (#12743171)
> You end up with a complicated dependency headache which hurts productivity.

>With a monorepo, you can statically identify all places where your library is used and update those automatically with tooling, or manually. You can also monitor where and how a piece of code is used across the company, etc.

Aka you have to figure out dependencies regardless of monorepo or not. Aka you need dependency management. Of course you could make the argument everyone has to use the latest greatest but you now have the possibility of changing one dependency and requiring an entire redeploy of the whole company.... I have seen what people do in these cases during desperate times... they copy the code and put it in their own project which sort of defeats the purpose and worse it is now not tracked (well I suppose if they have a good enough SCM comment it sort of is).

> If you then ensure that only commits get accepted that pass the relevant tests, you end up with a sane and working HEAD that's always up to date.

It depends on workflow. For some HEAD is what is actually deployed. If that is the case it is fairly difficult to achieve that goal with a giant monorepository if you have tons of teams doing microservices and deploying often.

Re:Mercurial 4.0 Sprint Notes by groovy2shoes on Wednesday October 19, 2016 @02:22AM (#12741535)
> When you split your libraries into separate projects, you have to start versioning them.

Or you can have your tooling use the convenient built-in versioning provided by the VCS. It's not called a "version control system" for nothing.

> And every update to the code requires other teams to then update their library/app to use the new version, and apps depending on that dependency... you get my drift.

This happens regardless of whether the codebase is organized as a monorepo or not. I've been subjected to monorepos at my previous and current employers, and we run into difficulties with this all the time. At least once a quarter, I get an e-mail from someone telling me that they updated such-and-such, and now the build fails in one of my projects. So I have to sit there and figure out wth they changed, why it's causing my code to fail the build, and how to fix it. Only once was the problem actually in my code (my Makefile, actually, which failed to bind some variable that was expected by the build system but wasn't documented anywhere, and of course the build system spits out a worthless error message, but I digress).

> You end up with a complicated dependency headache which hurts productivity.

Again, monorepos aren't immune from this, nor do polyrepos inherently suffer from it.

> With a monorepo, you can statically identify all places where your library is used and update those automatically with tooling, or manually. You can also monitor where and how a piece of code is used across the company, etc.

There is literally nothing preventing this from being doable with a polyrepo. In the case of updates, you may need to have the updated repos checked out, but it's not like you can only ever have one repo checked out at a time.

> If you then ensure that only commits get accepted that pass the relevant tests, you end up with a sane and working HEAD that's always up to date.

I've never worked anywhere on any project that disallowed commits that didn't pass the tests. It's typically been up to the committer to ensure that what they commit is acceptable. Furthermore, while my employers thus far have all been customers of AccuRev or Perforce (whose only real feature beyond what Subversion offers is that they cost lots of money), and have not personally had the pleasure of working with a DVCS at my day job, this workflow of "allow only test-passing commits" is wholly contrary to the "many small commits in quick succession" workflow afforded by DVCSs, which I contend is one of the biggest productivity boosts afforded by DVCSs (along with cheap branches and network unnecessity) and one of their main draws. And even so, it's still rather easy to maintain a sane and working HEAD with a DVCS (thanks to the cheap branches if nothing else).

If you find that you can't do anything you've listed here without a monorepo, that's a tooling issue rather than an organizational one. Frankly, I've never seen a large monorepo I felt was justified. Every single one was just baggage held over from a time long past when a single repo still made sense for the codebase.

Re:Mercurial 4.0 Sprint Notes by avar on Tuesday October 18, 2016 @01:38PM (#12736858)
As someone who works at another company with a big monorepo, briefly:

* Ability to change an API and all its users at the same time.

* Circular dependencies become a non-issue in a lot of cases where the would be if you vendor your dependencies.

* Even if you vendor your dependencies hunting for bugs is a lot easier, your bisect of a bug in a library will just come down the commit that upgraded it from 1.0 to 2.0 without a monorepo, with a monorepo you can bisect anything down to a specific commit.

* Code discoverability / mass edits are a lot easier, and integrate seamlessly into a lot more tools than if you need to manage multiple checkouts etc.

Re:Mercurial 4.0 Sprint Notes by hinkley on Tuesday October 18, 2016 @03:27PM (#12737846)
The big thing I miss about Subversion is that you could check out one subtree. On the biggest SVN project I worked on, we built 5 binaries from the same code tree, but only the leads and a couple of senior devs that worked on crosscutting concerns (ie, the people who would do the mass edits you mention) had the entire tree checked out. Everyone else had just the one or two modules they were working on.

The second biggest, we used just enough of a module system to enforce modularity by having a single code repo but separate compilation units. Your circular dependencies between subprojects would show up at compile time (or at least, on a clean build, as our CI machine did). Stopped a lot of obscure runtime issues and made people think about what they were trying to do.

Re:Mercurial 4.0 Sprint Notes by tux1968 on Tuesday October 18, 2016 @04:35PM (#12738413)
This post is about Mercurial with which i'm not familiar, but in Git you can checkout a subtree since version 1.7[1]. It's commonly called "sparse checkout".

[1] http://jasonkarns.com/blog/subdirectory-checkouts-with-git-s...

Re:Mercurial 4.0 Sprint Notes by glandium on Tuesday October 18, 2016 @06:22PM (#12739260)
But you still need to clone the whole thing. Sparse clones are theoretically possible, but aren't implemented AFAIK.
Re:Mercurial 4.0 Sprint Notes by emmelaich on Tuesday October 18, 2016 @08:41PM (#12740017)
You can do a shallow clone; see the --depth option.
Re:Mercurial 4.0 Sprint Notes by glandium on Tuesday October 18, 2016 @10:54PM (#12740648)
That reduces the history depth, but still gets the whole tree. Arguably, the history is much more interesting than having the entirety of the monorepo for many people.
Re:Mercurial 4.0 Sprint Notes by masklinn on Tuesday October 18, 2016 @01:37PM (#12736851)
> e.g. why doesn't FB use dependency management (or binary package management aka maven, npm, etc)? That is have multiple repositories and cut releases. Build tools to help cut and manage the releases and dependency graph of the releases.

Because it's a pain in the ass and a monorepo means you don't have to deal with that crap for internal code. It's especially important for refactorings or API changes where you can atomically perform a change at once rather than have to wait for the changes to trickle down across hundreds of repositories over days or weeks.

> potential coupling that can happen with a gigantic code base

The coupling is a feature not a bug. Decoupling is a mean not an end, and it has costs. If you don't need it you'd rather not pay it. Same as generic collections, if you don't need them there's nothing wrong with specific ones, quite the opposite.

Re:Mercurial 4.0 Sprint Notes by warbiscuit on Tuesday October 18, 2016 @04:06PM (#12738168)
Aside from all the other technical bits, one killer feature for my company is TortoiseHG (http://tortoisehg.bitbucket.org/). Unlike a number of the other Tortoise-* projects (which don't seem related actually), TortoiseHG is a really great cross-platform gui, written in python & qt -- runs just about everywhere, and makes working with mercurial uniformly easy.

The fact that instead of "hg ci" I can type "thg ci" and get a commit window with cherry-picking and meld integration right off the bat is really powerful. I've not seen any good git guis that come close to it's featureset.

Re:Mercurial 4.0 Sprint Notes by rjbwork on Tuesday October 18, 2016 @05:51PM (#12739048)
Yeah, THg is the absolute best DVCS GUI client I've had the pleasure of using. Everything else is awful and confusing in comparison IMO.

I even use it for Git with hggit sometimes.

Re:Mercurial 4.0 Sprint Notes by cmrdporcupine on Tuesday October 18, 2016 @01:29PM (#12736777)
Been at Google for coming up 5 years and I don't think I'm giving away any secrets when I say don't recall there being any performance issues with the mono repository.

And coupling is what is explicitly being sought, not rejected. The whole point is to build everything off head, keep head sane at all times, and avoid version dependency hell.

Re:Mercurial 4.0 Sprint Notes by pierrebai on Tuesday October 18, 2016 @02:30PM (#12737307)
Then maybe you can answer a few questions:

- Do you have zero external dependencies on 3rd party library not own by Google? These must still be managed anyway? How do you deal with project X is not ready to move to external library version N but project Y needs version N?

- What about release branches? If you need to intergrate a bug-fix in a sub-system, the magic mono-repo now means merging a fix is harder as it may depend on other unrelated changes all over teh repo? The laissez-faire attitude of not having to care about details in HEAD would seem to bite back in release branches.

Re:Mercurial 4.0 Sprint Notes by DannyBee on Tuesday October 18, 2016 @04:01PM (#12738126)
"- Do you have zero external dependencies on 3rd party library not own by Google? These must still be managed anyway? How do you deal with project X is not ready to move to external library version N but project Y needs version N? "

I own third party policy, so i can answer this. I'll stick to public info.

There are thousands of 3rd party libraries. The rules of the shared codebase is "third party libraries are not free headcount". You choose whether you use them or not (implicitly or explicitly). If you add a third party library, you get to maintain it, and stay within the support horizon of upstream. If you choose to use one, you get to stay up to date with the upgrades others make. If you need features not in the current version, you get to upgrade the library (and work with teams, who are not allowed to block you).

This is pretty much the only way to make it all work in practice (i'm aware of how it sounds. In practice, even upgrading stuff literally every google target depensd on takes a week. It's only stuff where folks let it go for 6 years [we now have better detection of out-of-date code] that become a problem to upgrade).

If you don't like it, don't use third party code.

Note that these rules make it just like any other code, because your problem is not specific to third party code.

Note that binary versioning, etc, is pretty much always a complete disaster in practice on a large scale

"the magic mono-repo now means merging a fix is harder as it may depend on other unrelated changes all over teh repo?"

This is rarely true in practice, because usually fixes are targeted.

Re:Mercurial 4.0 Sprint Notes by kyrra on Tuesday October 18, 2016 @03:49PM (#12738029)
From the ACM link someone else provided:

> An area of the repository is reserved for storing open source code (developed at Google or externally). To prevent dependency conflicts, as outlined earlier, it is important that only one version of an open source project be available at any given time. Teams that use open source software are expected to occasionally spend time upgrading their codebase to work with newer versions of open source libraries when library upgrades are performed.

So if an external library has to be updated, the entire codebase must be migrated all at once. Then Rosie is used to split the change into a lot of smaller changes (to be reviewed by all affected teams). Once all the smaller changes are LGTM'ed, it's submitted all at once.

Re:Mercurial 4.0 Sprint Notes by z3t4 on Tuesday October 18, 2016 @03:09PM (#12737674)
I think its insane not to have dependencies and third party code in your scm. Although it seems facebook plan to move those to a package manager due to performance problems and libs touching ten thousand files in a path release.
Re:Mercurial 4.0 Sprint Notes by agentgt on Wednesday October 19, 2016 @08:46AM (#12743234)
Yes but the question is where do you stop. Are you going to check in GNU Make and the compilers as well? How about whole operating systems?

I also would say "only check in what you change" is a simpler rule than "check in all dependencies needed".

I understand company source code needs to be different because of security reasons and convenience but should it really be that much different than any OSS project (which I guarantee would be pissed if you checked in all your dependencies).

> I think its insane not to have dependencies and third party code in your scm

Consequently there are some that might think the complete opposite (I wouldn't assign insanity but I would say it probably in most cases is not the right thing to do).

Re:Mercurial 4.0 Sprint Notes by z3t4 on Thursday October 20, 2016 @02:44AM (#12750169)
It all depends. Google might want to keep track of the full stack, whereas your average small software company will quickly adapt when their platform pulls the rug.
Re:Mercurial 4.0 Sprint Notes by cmrdporcupine on Tuesday October 18, 2016 @02:33PM (#12737334)
I am not sure how much of that I can answer in detail, sorry. But yes we have working strategies for dealing with both situations.
Re:Mercurial 4.0 Sprint Notes by tonfa on Tuesday October 18, 2016 @03:08PM (#12737671)
http://m.cacm.acm.org/magazines/2016/7/204032-why-google-sto... has a lot of answers (was published last summer).
Re:Mercurial 4.0 Sprint Notes by babyrainbow on Tuesday October 18, 2016 @08:38PM (#12740001)
May be I am not understanding the problem correctly, but I wonder why you couldn't use separate repos, and build a script to record the current commit hash of every other repo when you make a commit in any one of the repo.

So every commit in every repo has a map that maps the repo path to a commit hash.

You can use this info to sync the versions of other repos when you update/checkout a version in any of the repos. And all of these can be scripted.

Re:Mercurial 4.0 Sprint Notes by jcranmer on Tuesday October 18, 2016 @08:41PM (#12740022)
Why would you resort to a set of hacky scripts that everyone who wants to use the repository must follow when you can just let the VCS itself manage everything for you?
Re:Mercurial 4.0 Sprint Notes by babyrainbow on Tuesday October 18, 2016 @10:56PM (#12740660)
>Why would..

Because then you can keep the repos separated. Not saying that you should. But if this is the only problem that forces you to use a mono repo, then I am just asking if that problem can be solved in this manner?

>hacky scripts that everyone who wants to use the repository must follow when you can just let the VCS itself manage everything for you?..

What hacky scripts? Do you think they (google, facebook etc) don't have enough resources to build a 'non hacky' script for what ever they needs done?

Re:Mercurial 4.0 Sprint Notes by steveklabnik on Tuesday October 18, 2016 @01:11PM (#12736571)
I haven't worked on a monorepo, but I will say that Facebook _does_ use at least some of these tools, apparently: enough so that they created a whole replacement for the npm client, yarn.

From talking to some companies that use monorepos that want to use Rust, they _do_ use these kinds of tools, but want support for it in the tool. Firefox, for example, is pretty much a monorepo, and we built tooling in Cargo to help.

I think that part of it is that "monorepo" can mean slightly different things. Does it only contain your companies' code, or also your dependencies? Do you modify those dependencies in-repo?

Re:Mercurial 4.0 Sprint Notes by agentgt on Tuesday October 18, 2016 @01:18PM (#12736658)
Yeah I'm not even sure I understand what FB's monorepository is so it is probably unfair critique.

I can definitely see an "app" (Firefox) as a monorepo but not giant web companies (particularly with the surge of microservices).

BTW Cargo is fantastic (and I'm fairly picky on package management / build tools).

Re:Mercurial 4.0 Sprint Notes by adrianratnapala on Tuesday October 18, 2016 @03:01PM (#12737600)
Regarding performance: these mono-repo tools work differently and are optimised for their workload.

Regarding why the hell you would do it: In a corporate setting you tend to throw all kinds of st into your repo. In my workplace that includes, network configurations, 200 MB binary tools, usernames (and who knows, passwords too probably).

We do this because we want all the different things, including external dependencies and "ops stuff" to be syncronised. That would be totally insane in the open source world where you are releasing to the public -- but it seems corporations can make it work internally.

Re:Mercurial 4.0 Sprint Notes by jordigh on Tuesday October 18, 2016 @03:47PM (#12738008)
Re:Mercurial 4.0 Sprint Notes by gcp on Tuesday October 18, 2016 @01:18PM (#12736656)
OSSing certain parts is certainly an issue, but what do you mean with "potential coupling" being an issue? That's exactly the advantage, right?
Re:Mercurial 4.0 Sprint Notes by agentgt on Tuesday October 18, 2016 @01:23PM (#12736712)
I suppose you could make the argument the other way but my thoughts are that by having the code all together it maybe confusing what is public API (ie interfaces / contracts) and what is not.

I know personally that I have had to move components into separate projects (not necessarily repositories but compile units) to avoid developers from accessing things they shouldn't (e.g. access the database directly instead of going through a different layer).

Separate repositories would be enforcing it to a greater extent. But like I said earlier I suppose I could see it the other way (ie visibility and thus prevention).

Re:Mercurial 4.0 Sprint Notes by barkingcat on Tuesday October 18, 2016 @01:44PM (#12736905)
I don't use version control systems to any extent but it seems like your developer problem is an issue of team-coherence and code review, and not a matter that your vcs should/need to enforce.

Even in a distributed team like linux kernel development, code maintainers have the ability to say "this is how we (as core maintainers) want you to access certain data structure. If we catch you doing anything else illegally, your code won't be accepted, and we will keep rejecting your code until you adhere to our coding style and api design."

Re:Mercurial 4.0 Sprint Notes by agentgt on Tuesday October 18, 2016 @02:27PM (#12737272)
While I appreciate the "code review should be improved" it is fairly easy to accidentally import something you should not have in many IDEs (the merge the class path) as well as other accidents. These issues along with many others including code formatting can be automated.

Separate projects allow less namespace collision accidents.

You could achieve all of this with monorepo but requires proper tooling.

Re:Mercurial 4.0 Sprint Notes by amluto on Wednesday October 19, 2016 @12:39AM (#12741106)
As an open-source example, the Linux kernel is a mono repository. It makes refactoring easier, and it makes it much easier to drop support for old (or not so old) unused features.
Re:Mercurial 4.0 Sprint Notes by agentgt on Wednesday October 19, 2016 @08:21AM (#12743102)
The Linux Kernel and Firefox (mentioned earlier) are not really good examples. I would put those in a single repository as I would imagine most people would.

But a whole company as big as FB and Google using a single tree seems like an incoherent nightmare without proper (somewhat proprietary) tooling (filtering of logs, branches, tags, etc).

Re:Mercurial 4.0 Sprint Notes by jahewson on Tuesday October 18, 2016 @08:24PM (#12739927)
> There are even plugins and shell scripts that will make Mercurial act like a mono repository for many small repositories

Seems kinda backwards to me, why not project portions of a monorepo to behave like individual repositories? That way you get atomic commits too.

Re:Mercurial 4.0 Sprint Notes by agentgt on Wednesday October 19, 2016 @08:12AM (#12743043)
> Seems kinda backwards to me, why not project portions of a monorepo to behave like individual repositories? That way you get atomic commits too.

But on the other hand you now need custom scripts to filter out other projects that are completely irrelevant to you. Just the .hgtags file alone must be a 10k line nightmare... or maybe the big boys just don't deploy/release often. It is a lot more tooling than I think you might think.

Linux and Firefox having a monorepository is not the same as a whole company as big as google and facebook having a single repository for all of their projects.

I'm not even sure if these companies really are doing that. The use of monorepository could just mean a single place of storage but maybe they do allow a couple of trees (or maybe not).

Re:Mercurial 4.0 Sprint Notes by smegel on Tuesday October 18, 2016 @06:08PM (#12739170)
Maybe Docker is the solution.

Your "working state" is now in an image (maybe a Docker file that checks out particular revisions of different repos), so putting everything into one giant VC repo is not necessary.

A versioned Dockerfile that says "this is how we built SystemX at version 1.5" is much better than doing it in Git, as it also covers how the underlying server was built, while a giant git repo might have every application you need at the correct version, but won't say anything about the server it is deployed to.

Re:Mercurial 4.0 Sprint Notes by jahewson on Tuesday October 18, 2016 @08:20PM (#12739910)
The advantage of a monorepo is that you have atomic commits. What you're suggesting does not provide that.