An example of utility of monorepos is things like automation--if you change, for example, how you publish packages, and you need to maintain multiple stable branches, it is immensely useful to keep the automation steps in the same repository as code. If you don't, your automation repository then looks like
if version < 31:
step_a()
step_b()
step_c()
elif version < 35:
step_a()
step_bv2()
step_d()
else:
...
which quickly grows unmaintainable at scale.That said I have no idea how Netflix does source control, only that they use BitBucket/Stash on-prem and that supports both mercurial and git.
The only immensely dangerous thing that can happen is if you drop your package repository or change formats of the repository which rarely happens.
And every update to the code requires other teams to then update their library/app to use the new version, and apps depending on that dependency... you get my drift.
You end up with a complicated dependency headache which hurts productivity.
With a monorepo, you can statically identify all places where your library is used and update those automatically with tooling, or manually. You can also monitor where and how a piece of code is used across the company, etc.
If you then ensure that only commits get accepted that pass the relevant tests, you end up with a sane and working HEAD that's always up to date.
Of course there are lot's of drawbacks as well.
But Google and FB seem to have concluded that this is the better approach for them.
>With a monorepo, you can statically identify all places where your library is used and update those automatically with tooling, or manually. You can also monitor where and how a piece of code is used across the company, etc.
Aka you have to figure out dependencies regardless of monorepo or not. Aka you need dependency management. Of course you could make the argument everyone has to use the latest greatest but you now have the possibility of changing one dependency and requiring an entire redeploy of the whole company.... I have seen what people do in these cases during desperate times... they copy the code and put it in their own project which sort of defeats the purpose and worse it is now not tracked (well I suppose if they have a good enough SCM comment it sort of is).
> If you then ensure that only commits get accepted that pass the relevant tests, you end up with a sane and working HEAD that's always up to date.
It depends on workflow. For some HEAD is what is actually deployed. If that is the case it is fairly difficult to achieve that goal with a giant monorepository if you have tons of teams doing microservices and deploying often.
Or you can have your tooling use the convenient built-in versioning provided by the VCS. It's not called a "version control system" for nothing.
> And every update to the code requires other teams to then update their library/app to use the new version, and apps depending on that dependency... you get my drift.
This happens regardless of whether the codebase is organized as a monorepo or not. I've been subjected to monorepos at my previous and current employers, and we run into difficulties with this all the time. At least once a quarter, I get an e-mail from someone telling me that they updated such-and-such, and now the build fails in one of my projects. So I have to sit there and figure out wth they changed, why it's causing my code to fail the build, and how to fix it. Only once was the problem actually in my code (my Makefile, actually, which failed to bind some variable that was expected by the build system but wasn't documented anywhere, and of course the build system spits out a worthless error message, but I digress).
> You end up with a complicated dependency headache which hurts productivity.
Again, monorepos aren't immune from this, nor do polyrepos inherently suffer from it.
> With a monorepo, you can statically identify all places where your library is used and update those automatically with tooling, or manually. You can also monitor where and how a piece of code is used across the company, etc.
There is literally nothing preventing this from being doable with a polyrepo. In the case of updates, you may need to have the updated repos checked out, but it's not like you can only ever have one repo checked out at a time.
> If you then ensure that only commits get accepted that pass the relevant tests, you end up with a sane and working HEAD that's always up to date.
I've never worked anywhere on any project that disallowed commits that didn't pass the tests. It's typically been up to the committer to ensure that what they commit is acceptable. Furthermore, while my employers thus far have all been customers of AccuRev or Perforce (whose only real feature beyond what Subversion offers is that they cost lots of money), and have not personally had the pleasure of working with a DVCS at my day job, this workflow of "allow only test-passing commits" is wholly contrary to the "many small commits in quick succession" workflow afforded by DVCSs, which I contend is one of the biggest productivity boosts afforded by DVCSs (along with cheap branches and network unnecessity) and one of their main draws. And even so, it's still rather easy to maintain a sane and working HEAD with a DVCS (thanks to the cheap branches if nothing else).
If you find that you can't do anything you've listed here without a monorepo, that's a tooling issue rather than an organizational one. Frankly, I've never seen a large monorepo I felt was justified. Every single one was just baggage held over from a time long past when a single repo still made sense for the codebase.
* Ability to change an API and all its users at the same time.
* Circular dependencies become a non-issue in a lot of cases where the would be if you vendor your dependencies.
* Even if you vendor your dependencies hunting for bugs is a lot easier, your bisect of a bug in a library will just come down the commit that upgraded it from 1.0 to 2.0 without a monorepo, with a monorepo you can bisect anything down to a specific commit.
* Code discoverability / mass edits are a lot easier, and integrate seamlessly into a lot more tools than if you need to manage multiple checkouts etc.
The second biggest, we used just enough of a module system to enforce modularity by having a single code repo but separate compilation units. Your circular dependencies between subprojects would show up at compile time (or at least, on a clean build, as our CI machine did). Stopped a lot of obscure runtime issues and made people think about what they were trying to do.
[1] http://jasonkarns.com/blog/subdirectory-checkouts-with-git-s...
Because it's a pain in the ass and a monorepo means you don't have to deal with that crap for internal code. It's especially important for refactorings or API changes where you can atomically perform a change at once rather than have to wait for the changes to trickle down across hundreds of repositories over days or weeks.
> potential coupling that can happen with a gigantic code base
The coupling is a feature not a bug. Decoupling is a mean not an end, and it has costs. If you don't need it you'd rather not pay it. Same as generic collections, if you don't need them there's nothing wrong with specific ones, quite the opposite.
The fact that instead of "hg ci" I can type "thg ci" and get a commit window with cherry-picking and meld integration right off the bat is really powerful. I've not seen any good git guis that come close to it's featureset.
I even use it for Git with hggit sometimes.
And coupling is what is explicitly being sought, not rejected. The whole point is to build everything off head, keep head sane at all times, and avoid version dependency hell.
- Do you have zero external dependencies on 3rd party library not own by Google? These must still be managed anyway? How do you deal with project X is not ready to move to external library version N but project Y needs version N?
- What about release branches? If you need to intergrate a bug-fix in a sub-system, the magic mono-repo now means merging a fix is harder as it may depend on other unrelated changes all over teh repo? The laissez-faire attitude of not having to care about details in HEAD would seem to bite back in release branches.
I own third party policy, so i can answer this. I'll stick to public info.
There are thousands of 3rd party libraries. The rules of the shared codebase is "third party libraries are not free headcount". You choose whether you use them or not (implicitly or explicitly). If you add a third party library, you get to maintain it, and stay within the support horizon of upstream. If you choose to use one, you get to stay up to date with the upgrades others make. If you need features not in the current version, you get to upgrade the library (and work with teams, who are not allowed to block you).
This is pretty much the only way to make it all work in practice (i'm aware of how it sounds. In practice, even upgrading stuff literally every google target depensd on takes a week. It's only stuff where folks let it go for 6 years [we now have better detection of out-of-date code] that become a problem to upgrade).
If you don't like it, don't use third party code.
Note that these rules make it just like any other code, because your problem is not specific to third party code.
Note that binary versioning, etc, is pretty much always a complete disaster in practice on a large scale
"the magic mono-repo now means merging a fix is harder as it may depend on other unrelated changes all over teh repo?"
This is rarely true in practice, because usually fixes are targeted.
> An area of the repository is reserved for storing open source code (developed at Google or externally). To prevent dependency conflicts, as outlined earlier, it is important that only one version of an open source project be available at any given time. Teams that use open source software are expected to occasionally spend time upgrading their codebase to work with newer versions of open source libraries when library upgrades are performed.
So if an external library has to be updated, the entire codebase must be migrated all at once. Then Rosie is used to split the change into a lot of smaller changes (to be reviewed by all affected teams). Once all the smaller changes are LGTM'ed, it's submitted all at once.
I also would say "only check in what you change" is a simpler rule than "check in all dependencies needed".
I understand company source code needs to be different because of security reasons and convenience but should it really be that much different than any OSS project (which I guarantee would be pissed if you checked in all your dependencies).
> I think its insane not to have dependencies and third party code in your scm
Consequently there are some that might think the complete opposite (I wouldn't assign insanity but I would say it probably in most cases is not the right thing to do).
So every commit in every repo has a map that maps the repo path to a commit hash.
You can use this info to sync the versions of other repos when you update/checkout a version in any of the repos. And all of these can be scripted.
Because then you can keep the repos separated. Not saying that you should. But if this is the only problem that forces you to use a mono repo, then I am just asking if that problem can be solved in this manner?
>hacky scripts that everyone who wants to use the repository must follow when you can just let the VCS itself manage everything for you?..
What hacky scripts? Do you think they (google, facebook etc) don't have enough resources to build a 'non hacky' script for what ever they needs done?
From talking to some companies that use monorepos that want to use Rust, they _do_ use these kinds of tools, but want support for it in the tool. Firefox, for example, is pretty much a monorepo, and we built tooling in Cargo to help.
I think that part of it is that "monorepo" can mean slightly different things. Does it only contain your companies' code, or also your dependencies? Do you modify those dependencies in-repo?
I can definitely see an "app" (Firefox) as a monorepo but not giant web companies (particularly with the surge of microservices).
BTW Cargo is fantastic (and I'm fairly picky on package management / build tools).
Regarding why the hell you would do it: In a corporate setting you tend to throw all kinds of st into your repo. In my workplace that includes, network configurations, 200 MB binary tools, usernames (and who knows, passwords too probably).
We do this because we want all the different things, including external dependencies and "ops stuff" to be syncronised. That would be totally insane in the open source world where you are releasing to the public -- but it seems corporations can make it work internally.
I know personally that I have had to move components into separate projects (not necessarily repositories but compile units) to avoid developers from accessing things they shouldn't (e.g. access the database directly instead of going through a different layer).
Separate repositories would be enforcing it to a greater extent. But like I said earlier I suppose I could see it the other way (ie visibility and thus prevention).
Even in a distributed team like linux kernel development, code maintainers have the ability to say "this is how we (as core maintainers) want you to access certain data structure. If we catch you doing anything else illegally, your code won't be accepted, and we will keep rejecting your code until you adhere to our coding style and api design."
Separate projects allow less namespace collision accidents.
You could achieve all of this with monorepo but requires proper tooling.
But a whole company as big as FB and Google using a single tree seems like an incoherent nightmare without proper (somewhat proprietary) tooling (filtering of logs, branches, tags, etc).
Seems kinda backwards to me, why not project portions of a monorepo to behave like individual repositories? That way you get atomic commits too.
But on the other hand you now need custom scripts to filter out other projects that are completely irrelevant to you. Just the .hgtags file alone must be a 10k line nightmare... or maybe the big boys just don't deploy/release often. It is a lot more tooling than I think you might think.
Linux and Firefox having a monorepository is not the same as a whole company as big as google and facebook having a single repository for all of their projects.
I'm not even sure if these companies really are doing that. The use of monorepository could just mean a single place of storage but maybe they do allow a couple of trees (or maybe not).
Your "working state" is now in an image (maybe a Docker file that checks out particular revisions of different repos), so putting everything into one giant VC repo is not necessary.
A versioned Dockerfile that says "this is how we built SystemX at version 1.5" is much better than doing it in Git, as it also covers how the underlying server was built, while a giant git repo might have every application you need at the correct version, but won't say anything about the server it is deployed to.
The benefits of having one repository do not seem to be worth the serious performance issues as well as potential coupling that can happen with a gigantic code base as well also making much more difficult to OSS certain parts.
e.g. why doesn't FB use dependency management (or binary package management aka maven, npm, etc)? That is have multiple repositories and cut releases. Build tools to help cut and manage the releases and dependency graph of the releases.
There are even plugins and shell scripts that will make Mercurial act like a mono repository for many small repositories (I use it for our own Java Maven code base).
I must be missing some killer features and would love to see it in action (FB major repository).