The company I work for has recently transitioned from many small repos to a single monorepo. I can tell you that the development experience is MUCH better. 99% of issues required cross-repo coordination, which was a nightmare, and thing could easily get out of sync.

With a monorepo, you don‘t need to think about which commits work with which other commits. One commit ID is a full description of every subcomponent.

I don't really understand how that creates fewer bottlenecks. It may save communication overhead because every dev can spot integration issues on their own, but at the cost of monstrously complex and time-consuming build-and-test pipelines. The microservice approach just requires some kind of service contract and meaningful release versions. That's always been a very manageable cost in my experience.

> just requires some kind of service contract and meaningful release versions.

I've just switched to a company with many tiny repos and IMO it's a huge hassle. There's no automated integration testing, and manual integration testing is a huge pain to set up.

I don't even know how I'd create integration tests that run as part of the CI. What version would I use? If you need to change both sides (very common), coordinating the commits and releases is very painful.

The "monstrously complex" build-and-test pipelines are a significant cost, but the alternative is higher release failure rate and moving slower overall IMO.

In my experience the Uber mono repo was a giant velocity killer. Go tooling just completely choked, and at the time they had tons of terrible hacks around go modules by having this weird system that required your mono repo to be in GOPATH, further breaking even more tooling.

No clue if that’s fixed today but it soured the idea of monorepos for me, and that’s not incorporating how often submitqueue would go down.

The sharing of service contracts and their shared dependencies is a difficult problem to solve by itself. There's companies trying to make money off of the problem (buf). Monorepos solve the problem out of the box.

If 99% of issues required cross-repo coordination, then those repos by definition should not have been distinct from each other. Splitting code into separate repos only makes sense if changes can be shipped in isolation.

1 repo == 1 bounded context == 1 isolated unit of deployment.

No true scotsman would split up repos wrong.

Then any time you have fan-shaped dependencies, you need a monorepo.

Which, well, the Uber people clearly have since these big changes exist.

> 1 repo == 1 bounded context == 1 isolated unit of deployment.

The most obvious counterpoint: many systems of record need to scale command operations (writes) and query operations (reads) separately.

So you absolutely would have separate programs, ASGs etc for those two roles

How is this a counterpoint?

They wouldn’t have a single deployment but exist in one bounded context.

Oh, indeed, my mistake. A team can of course deploy its bounded context at whatever granularity it prefers. The important thing is just that that deployment schedule is unrelated to the deployment schedule of other teams/contexts.

Monorepos have the big advantage that cross-project changes can be made together in one PR.

Then why not call it just a repo and build a monolith?

I've seen people going crazy with this bullshit of monorepos to the point every single directory was a "package" when it could be just a plain module import.

If you want to do microservices, then putting everything back into a single repository and enforcing everyone to use the same version and every change to require upgrades and deploys across the board is totally backwards.

Either do a monolith and have that consistency, or do microservices and allow teams to follow their own rules as long as they keep APIs stable.

Nonorepos + microservices is just a demonstration of everything that's wrong in the technical aspects of our industry. Just applying absolutely everything you read about without even considering if it might be better or not for your specific use case.

The way you store your source code doesn't have much to do with how your system is deployed. It's the same way you can have a closet in which you store clothing for many purposes; clothing for cold weather, clothing for hot weather, clothing for formal occasions, clothing for specific recreational activities, et cetera. You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree.

The way you store your source code is 100% tied to how your system is deployed. Repos are the thing that define atomic changes at the source level. If a change at your business level requires atomic changes at the source level between X Y and Z, and X Y and Z aren't in the same repository, then you done played yourself.

> You can store code for different purposes in the same source tree. You can deploy something that is built from just one part of the source tree.

At the risk of sounding memetic, the question is not so much "Can you?" but "Should you?". Should you store a monolithic application across multiple repositories? Should you store a distributed application in a monolithic repository?

I agree with @likortera that you should not. And that's because...

> The way you store your source code doesn't have much to do with how your system is deployed.

I disagree with this statement.

Your architecture (monolithic vs distributed) imposes certain assumptions on other aspects of your distribution pipeline. Your workflow (in this discussion "how you store your source code") should support these assumptions, not hinder them.

For example, one benefit of microservices is that cross-functional teams can develop independently of each other. And yet one cited advantage of monorepos is that everyone is on the same version of dependencies all the time. In short, your teams are not independent after all.

Note that I'm keeping the example extremely generic to illustrate this inconsistency, a conflict of interest if you will, that I see people commit in this topic, because in my experience, these questions are not purely technical but involves product/business factors as well. Maybe for most of the people (operative emphasis on "MAYBE", because who am I to judge you), the discussion they need to have first is whether or not they are using the right architecture for their product in the first place.

If you choose to have a microservices architecture, you have to live with the fact that your teams/services will operate at different cadences. If you feel the need to impose a One True Library Version All the Time, then go for a monolithic architecture, and store your code in the same way.

Two things.

One, teams that share a dependency version can still develop independently; they just share something in common. They already likely share other things in common: deployment target OS, cloud platform and services, shared authN/authZ frameworks, etc.

Two, a monorepo is just the SCM mechanism. As the parent was describing, it doesn't prescribe anything other than the code storage location and how branching, committing, etc. works. Yes, a lot of organizations prefer having a single version rule in their individual monorepos, but nothing about monorepos in general makes this a requirement. You can use multiple versions of the same dependency and still gain advantages from the single commit benefits, and even famous instances like Google's have exceptions where this is the case.

> a monorepo is just the SCM mechanism ... it doesn't prescribe anything other than the code storage location and how branching, committing, etc. works

That's not really the case in practice. When people decide to choose mono vs multi based on their benefits, it's become a workflow philosophy in itself. If you choose a monorepo approach but use multiple versions of the same (in-house) dependency across components, you are just opening yourself up for a world of confusion. Sure, you can do it, but should you? Why choose a monorepo structure if you won't take advantage of its benefits?

> teams that share a dependency version can still develop independently; they just share something in common

My point about team independence doesn't mean they should not share anything at all. But rather, they now _update_ together at the same pace because the "atomic commit" that updated a dependency also updated my team's usage of said dependency, for better and for worse. My team might have a reason not to update just yet.

Spot on!

One of the companies I worked for, had the brilliant idea of putting EVERYTHING related to UI/frontend in a monorepo, where almost every single file or two were a different package. I used to joke there were more package.json files than actual js files (it was almost true).

I spent months saying this thing was a terrible idea. Nonetheless the "frontend infrastructure" folks wanted to do some CV padding and play with Lerna and their SV friend's cloud CI service startup, so they went ahead with it.

Months later the big problems started, among which one of the main ones was that they were pushing through every team's throats updates, breaking the product/features those teams were working on, disrupting their roadmaps, accusing each other's of low test coverage, doing hacks and workarounds, shit tons of crazy CI scripts for all the corner cases, much longer deploy times, most dev environments were a lot slower, deployment issues because now we had to deploy several different projects at the same time, etc, and of course not being able to upgrade to latest React because some team in the corner had an issue with it and they didn't have the time at the moment to fix it.

How did they solve all of this? In the span of 3 to 4 months they left the company. All four of them. Leaving behind an incredible amount of technical debt and nearly every frontend team totally fucked up.

What irks me is that some of these guys are pretty popular "youtubers" and spend their days giving talks of how great their work with monorepos and "frontend infra" is. They don't tell the messes they've caused of course.

Monorepos might be great if you're Google and have the resources and talent to do it right. For most companies out there, it is just creating a centralized problem that will eventually block everyone.

I'm a big proponent of monoliths, specially while you're not a > 300 person company. But if you're splitting your teams and services, then agree on APIs, don't break them, and let each team follow their own pace, with their own tools, and with their own schedules and preferences. Otherwise stick to the good ol' monolith and just separate things into modules/imports/whatever.

At my current employer, the main application has a "plugins" system with a very flexible and stable API. Every team around is just building "plugins" that can be installed into the main monolith, depending on each customer needs. This works fantastically well for a company with more than 1k engineers. No monorepos, no coupling, no interdependencies between teams, no parallel deploys, and each team manages their own destiny more or less.

Microservices, monorepos, monoliths etc are all outcomes of applying different tradeoffs to the 4+1 Views of Architecture [1].

The development view need not be connected to the deployment view at all, so unnecessarily coupling them can lead to worse outcomes. On the other hand, having the ability to couple them initially _and decouple them again in future_ works wonders for scaling.

I am consistently amazed that people with opinions on software architecture do not seem to recognise this seminal paper on the topic, or have the ability to re-synthesise it into tactics.

[1]: https://www.cs.ubc.ca/~gregor/teaching/papers/4+1view-archit...

Why should the system for managing source code have anything to do with the architecture of the deployed code? This coupling seems arbitrary to me.

This is almost the only advantage. A related "advantage" is forced upgrades when dependencies change - since you can't have code depending on different versions of in-repo libraries and services, client code needs to adapt or die.

The more clients a library or service has, the more expensive this is, and it's an ongoing maintenance cost for every client. Since people changing the service don't feel the full pain of this maintenance cost, changes keep on happening, and eventually clients get culled because it's too expensive to keep on maintaining them all.

From the outside, this looks like the company abandoning venerable but still working product, and makes people scratch their heads, wondering why.

You can have a monorepo without having all the modules share the dependencies. For example you can have multiple go.mod files in a single monorepo.

People often conflate these two things because often teams actually want to have centralized dependencies (so that you're forced to update or die as you said). If that doesn't work for you you can choose to have modules (or groups of modules) keep their independent set of dependencies, all while keeping the code in the monorepo.

This is the correct answer. Putting all integration changes in a single PR improves CI test coverage and is easier to verify in a code review.

This really begs the question why Github aren't working on cross-repo PR support - or anyone. Everytime mono-repos come up, cross-repo PRs are the reason given as to why they're needed - why can't we get this supported well?

That’s not just a GitHub issue, that would require serious UX work on the authoring side to make such “commits” that are actually commits in multiple repos at the same time. Effectively this is recreating a monorepo (maybe with something like for submodules?)

I don't think it's recreating a monorepo. What you're looking for is something like a DB transaction where all the merges in different repos succeed or everything gets rolled back.

Maybe I am missing something: to do that you need some kind of “meta-repository” linking commits between all the different sub repos in some order, no? If that happens for a significant portion of the commits in a repo, isn’t that basically equivalent to a mono repo with git submodules or something?

Well a meta repo isn't a mono repo. In that situation you can still update individual repos. Analogous to running a DB operatioj without a transaction.

But that said, no I don't think you need a meta repo. You need something. Call it a Change set. MVP would be say merge commits in 4 repos. If any fail then revert. If they all succeed then deploy in a specified order.

Sourcegraph does a decent job of this with their “batch changes” feature

Large cross project changes mean you have a monolithic project. One project split over multiple repos, not multiple projects. If you're separate projects were really separate they'd maintain API compatibility and upgrade paths (eg. semantic versioning). SaaS and microservice architectures require this to maintain separation but most organizations lack the discipline and slowly revert back to a monolith as they are faster to develop (until they aren't).

Pros

* Breaking changes can be done at once. Very helpful for runtime deps.

* No chance that a repo is out of date.

* Upgrades are atomic (may be hard to test a system in a half state).

While you’re deploying, production is going to be in a half-upgraded state (or maybe half rolled-back!), so it’s pretty important to be able to test that.

This depends on the deployment model. I am curious, how does your release and deployment looks like?

Release is a privileged tool pushing to Git. Deployment is incremental across a small but growing number of containers running a microservice, so both old and new code might be running concurrently for up to a couple of hours. Percentage experiments are pretty common, which means both old and new paths actually have to work in the same commit.

A well factored system should not have cross-project changes. Individual projects should be able to upgrade dependencies separately.

You can use this argument against anything you don't like.

'A well-designed system should not have X therefore you don't need Y.'

When 'A well-designed system should not have X' is just a matter of opinion, or doesn't acknowledge trade-offs that might make X the better option than Z, then this isn't a useful argument against Y.

yeaaah, I used to hold this view and drove a team - hard - to essentially pull apart a monorepo into different components with clear contract boundaries. It was by far one of my greatest errors in professional judgement.

at the risk of coming up with a contrived example: let's say you own a service that need to deserialize a datetime in a request in a format you don't currently support. assuming you own the stack, you need to a) update your date library b) update your webserver stack c) possibly update an intermediate webserver stack that includes primitives like logging, telemetry, tracing, auth, service discovery and d) your actual service.

If a->d are all independent, separate components, you have to orchestrate those changes through 4 separate repositories. And god forbid something you did at the lowest point in the stack is completely unworkable higher up.

There's all sorts of rocket science you could do to orchestrate these changes, but it ends up being contrived and edgecasey.

Most of the pain from monorepos can also be addressed with a dash of rocketscience (see:bazel), but the end model tends to have

a) have an easier mental model for the user

b) allow for consolidation of infrastructure work. Ie, your build/ci/language tooling teams can focus their efforts on one place

c) can coordinate changes across the entire stack within one field of view

d) can coral some of the worst, disparate instincts of a growing engineering org (ie, tons of teams optimizing for local maximas without internalizing knock-on effects).

e) fewer weird edgecases.

Its important to avoid deep dependency chains when doing this, yes. I usually recommend a "framework" multi-package repo and then multiple "application" / "service" repos.

I like having examples, even contrived ones, but I'm not sure I understood this one. Can you elaborate on what you mean? Is it about adding support for a new serialization format for dates in requests to a service? Why would this affect the webserver stack and logging/telemetry/tracing/auth primitives?

I find that a lot of organizations have really strange thoughts on how to factor things into separate microservices and libraries. Usually I approach by asking the following question: if this was an open source-library or service (e.g. like elasticsearch), would you use it? If not, then its probably not a great candidate for a separate thing - lets try and come up with something else.

One way to handle CI/CD is using standardized pipelines e.g. you tag your repo with a tag `app:node` or `lib:js` and the github org pipeline scanner will find it and assign the standard `app:node` or `lib:js` pipeline to it.

A way that I like better but most tools unfortunatley don't support it yet is for the infra teams to publish libraries that are essentially functions taking some parameters and generating (standard) pipelines/configuration. Those can then be tracked together the same as other dependencies.

Splitting your code over many repositories massively increases the cost of refactoring, making it that less likely that your system will be "well factored".

Besides the "N pull requests" problem you now lose history whenever you move a file across repo boundaries, and you'll eventually have straggler projects staying on old versions for years - so switching to a new way of doing something essentially means supporting both versions forever. Code for common needs gets duplicated, or worse, split out into yet another repository and /then/ duplicated, because no one wants to check if any of 100 repositories rely on the buggy behaviour they want to fix.

I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/. I'm surprised to see so many comments summarily dismissing them, as if sanely managing thousands of smaller interdependent repositories doesn't require at least as much investment in custom tooling.

> Splitting your code over many repositories massively increases the cost of refactoring, making it that less likely that your system will be "well factored"

The project should stay a full monolith until the factoring is more clear.

> I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/.

There are two issues I have with this article. One of them is when it describes drawbacks of multiple repositories, its not specific enough e.g.

> That sounds like it ought to be straightforward, but in practice, most solutions are cumbersome and involve a lot of overhead.

The other is that it assumes you have Google scale of resources to throw at the problem. If thats the case you can make anything work / monorepos or polyrepos. The issue is that small-to-medium sized organizations will not be prepared to invest the amount of resources needed to keep a monorepo working well, as the org will largly need to rely on existing available (OSS) tools which often have poor monorepo support. (Bazelifying everything has a significant cost, bazel rules are often not generic enough to work with e.g. the variety of JS ecosystem tools)

Only if you break your code up on boundaries that don't reflect reality.

If one business change requires code changes to ~every module in your source tree, then of course microservices, and therefore separate repos per service or whatever, make no sense at all.

One of the examples given was a go compiler upgrade for all of their go code. It seems reasonable that some companies would want to have a single team investigate and perform this upgrade all at once.

Sure, each project team could upgrade independently, but the company that’s chosen a mono-repo for their go code is likely to desire to have a single team tackle this upgrade.

In theory.

In practice, you often end up with larger changes which are mostly janitorial. Bumping a version of a log library across the infrastructure, or a version of a serialization system. This simplifies dependency convergence and a best handled as if they are cross-project changes.

The other problem is that over time, good factoring tend to deteriorate. A large software project will invariably have people with different brains working on it, and they'll have different needs for the factoring. So the project naturally pushes itself toward situations where cross-project changes become a necessary thing.

We already have tools that handle this for us for OSS libraries, such as dependabot etc.

Good factoring can probably be quantified too: dependency chains should be shallow and graph connectivity should be low. The more dependants a module has, the more stable its API contract should be.

Well, almost everyone talking here about nonorepos why they are actually building are distributed monoliths. That's why they need changes cross all clients on the same PR.

can you provide more info here?

i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.

Changes will be handled just like external library updates.

i suspect your internal model of a monorepo is fairly disjoint with an actual implementation. you’re arguing for non-monorepo based version concepts, without giving space to why having everyone locked to “latest” has large cost savings long term for maintenance and security. without those constraints, a monorepo is a bad choice.

I've worked with both models, and the monorepo model has been far more demanding in terms of time and resources to maintain as you push past the limits of the available tools one by one. Standard build tools go out of the window almost immediately (lots of work to wrap everything with Bazel), then standard collaboration tools (GH) and so on. A dedicated tooling team becomes a must.

What helps maintain polyrepos sane

1. A healthy methodology for dependency management

Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful.

2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability.

3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.

The issue comes when business requirements don’t allow you to arbitrarily decide that your API should be deprecated. If it’s depended on by 10 teams who are doing higher impact customer facing work than you are, at some point you’re likely to hit a “no” when it comes to disrupting them.

How is that different in a monorepo? Would you be tasked in implementing the changes for those teams yourself? If so, why not just do that as PRs in their repos?

Good articulation of why things don’t quite play out that way within a company here https://rushjs.io/pages/intro/why_mono/

For that particular article, I would respond with the following:

1. I don't see how this is a problem specific to polyrepos. Have an "open PRs" link in the onboarding handbook that gives you a view of pull requests from all repos in the organization. GitHub automatically shows you notifications from all repos. If engineers still chose to focus on one or two repos after that, I'm not sure why.

- Have a (Grafana) dashboard where you can see the latest / newest stuff. Use standard GH tools you use for OSS, such as follows etc to keep up.

2. Don't prematurely split into multiple repos. "No monorepo" doesn't mean not having poly-package repos. It means thinking what the sensible (library or service) API boundary is - treating your projects as you would treat library / service development. In this case a separate repo with lib3, lib2 and lib1 sounds like a good way to go - at most one repo per orthogonal internal framework (e.g. core-react-components). Repo dependency chains should be as shallow as possible, and differenting between public and internal packages is important.

3. Help other teams upgrade. If you are responsible for repo A, once you publish a new version tagged appropriately with semver, use the dashboard to look at your dependants and work with them (or rather, for them) to upgrade. Think of your dependants as internal customers, and make sure you add enough value for them to justify the upgrade effort. Cultivate a culture that values updates.

4. There are other alternatives to `npm link` e.g. see `yalc` https://github.com/wclr/yalc

Another pet peeve of mine is that the real issues get lost when you try to generalize. The article attempts to do this but that makes it hard to evaluate its claims. The best way to evaluate (alternative) solutions is to take a more concrete example repo.

For scalability of this model I'll just point to the OSS community; individual maintainers often several dozen active repositories, but also they have an API contract worthy of a documentation website, versioning scheme and planned deprecation, and they typically avoid cross-project dependencies

> Individual projects should be able to upgrade dependencies separately.

No, that way lies sorrow and despair. There should only be one version of any dependency being used in your company. Any deviations should require like VP-level approvals or something.

It also helps keep people in sync with module and library versions. Without a monorepo, oftentimes what happens is that everybody ends up on different versions of libraries and modules. For every security update to a library, you'll have to campaign for everyone to pull that fix instead of everyone getting all fixes on every pull. It also forces dealing with incompatibilities between libraries early on when they happen instead of later when they are more entrenched and difficult to fix.

What I've seen in practice in smaller companies, is actually the other way around: everyone is afraid updating any library version because who knows what other team's work it might break. Theoretically, everyone must have extensive tests to cover everything, but in practice it's hard.

But when your team owns a repo, then at least the damage is contained within your team.

Google solves that problem with heavy NIH syndrome (its hard to get promoted by utilizing an external third-party lib, better develop your own), and writing tests, yes. And for those few third-party libraries that google still depends on, updating them is a big PITA.

It's not just NIH, after a certain size every external lib is a liability. You don't want to wait for some OSS maintainer to unblock your multi billion dollar business' issue.

If that were the case, you'd expect to see forking of 3rd party libraries rather than writing of new ones.

Most of the time you probably only use a small portion of the 3rd party library functionality so forking the whole thing and ripping out everything you don't use or maintaining it doesn't make much sense.

That can be handled with alerts and deprecations just like any other library dependencies

no it can’t, an alert won’t make a PM allocate time to “tech debt” and these alerts are rarely time free upgrades

If you have an issue handling tech debt due to PMs, you'll have that problem with other tech debt too; typically solved by having a better process to handle it

The problem isn't per se monorepo v multi-repo, it's ensuring that your lines of communication between components, ie your APIs, are shared reliably and coherently.

When teams take the stability & versioning of their APIs seriously, the need to use monorepos to share that info is greatly reduced. A multi-repo approach is perfectly feasible when all components are working to established APIs, which also alleviates the issues mentioned in the article.

I worked at Uber. The main arguments that I kept hearing were:

- easier to share and import common packages

- proto and thrift files are kept close to services and clients are updated globally automatically

- dependencies and go versions are managed globally and all services get the same security updates

- standardized build processes make it easier to manage large deployments

And honestly, after the initial repo download there were no visible downsides

How does not having a monorepo solve this problem? They would still need to compile and run the same code in order to run these tests between 1000 small repositories vs 1 large one, presumably they see a lot of value in end-to-end tests.

Is the suggestion just "an easier way to avoid this problem is to not have so many tests?"

If the change is specific to a single module, and the APIs exposed to other modules don't change (which can also be enforced with unit tests), you don't need to run the tests for every single repo for every change. Of course, one may argue that one could run module-specific tests in a monorepo as well. But that's not easy to do on a per-PR basis, so you just end up running the entire CI suite for every PR.

So you'd stop running tests for all of the calling code? It sounds like you have a higher level of confidence than I do that non-API changes can never break any downstream repositories in subtle ways.

that’s not actually what happens at Uber. They use bazel for the go monorepo which can accurately determine what modules are affected by a given change and only runs those tests.

They also use mostly homegrown CI tools alongside phabricator for code review, or at least they did while I was there.

I would argue it's impossible to know what you really changed without the integration tests.

But also, why is it any different why are changes at a repo level all that much easier to track than, say, inspecting changed files and running tests according to what changed?

We’re going to have monorepos until everyone tires of solving the problems they didn’t have before switching to this decade’s shiny object for workflow.

They used to have many small repos according to https://eng.uber.com/go-monorepo-bazel/ so perhaps there are advantages that outweigh slow CI.

MonolithFirst https://martinfowler.com/bliki/MonolithFirst.html

First yes, but it's not difficult to imagine Uber has outgrown their first architecture.

Uber was the opposite. In ~2017 after years of hyper growth they had over 10,000 repos supporting nearly as many micro-services. The move to monorepos (technically two, one for jvm stuff, one for go) came as a solution to the maintenance issues of this sprawl. I was there when a mandate came down to move your service into the monorepo.

I don’t understand how it’s possible to have close to 10,000 services. I worked on a project with 17 and found that to be very awkward at times (local development with k8s didn’t seem to be a solved problem, and we depended heavily on complex tests and blue/green deploys to be sure changes actually worked in production). But nearly 10,000 is insane.

How do you orchestrate that? I guess this is the kind of situation where you really do need container orchestration.

It's really easy to innocently slip into a culture of "new project? new service!" and at Uber's scale, that could easily hit 10k.

Yeah, I’m starting to realize my experience in software is even smaller scale than I thought.

Repos aren't one-to-one with running services. Lots of these could easily be bundled as modules to be included at build-time. A lot of them could also just be API services that provide a contract via REST or whatever. If you have enough internal users, you just have to maintain an SLA comparable to a public API.

Both google and Facebook have monorepos, and I think apple does too. I’m not sure about the others, but I wouldn’t be surprised if they did as well

Exactly. And probably what Apple, Google and Facebook do is just the exact opposite to what the other 99% of companies out there need.

Who on earth can think that what they do, companies with the engineering power as these companies have, has to be also good for it's 30 employee startup just boggles my mind. Not talking about Uber, I have no idea about them and what they do. But I worked for smaller startups just blindly following what they read google or Facebook do and immediately thinking that's the best thing to do too.

It's ridiculous.

Google also has projects like this: https://github.com/google/breakpad

It supports 5 platforms, but uses 4 completely different build systems, including 2 custom ones (3 if you count depot_tools). There is very little overlap between the platform versions, meaning it's effectively 5 different projects smashed together into a single folder, and pretty much no way to use them in a cross platform project without some serious work. There isn't even a basic abstraction over the similar callback APIs between the platforms, although that's not a huge deal because the effort to write a basic abstraction layer is nothing compared to the effort of getting to a point where you can actually use it in a cross-platform project.

It's also funny that one of the build systems is GYP, which is basically a reinvention of CMake, except it's only used for the Windows build even though it can generate projects for the other platforms. Also, the VS project generator for GYP has been broken for a while (simple typo, trying to import OrderedDict from the wrong module. There's a PR to fix it, hasn't been merged for some reason), so it doesn't even work. Beyond that, it's also broken because GYP forces treating all warnings as errors, with a whitelist of warnings, yet the latest version (since yesterday at least) fails to build (tested on VS2019) because there's a warning that isn't in the whitelist.

You could try to fork it and fix these issues, but depot_tools doesn't provide a way to change the clone URL for repos, meaning you need to dig through the source code and wrap it in your own script that interacts with the internal APIs to do a simple clone (hint: fetch.py has a 'run' method that you can call with a custom constructed 'spec' object, which is a dictionary where you can inject your own url; just look at the hard-coded spec object for breakpad as a starting point). If you don't use depot_tools, then you need to manually clone all of the dependencies in the project since they're not even set up as git submodules.

There's also no versioning scheme whatsoever. Depot_tools seems to automatically checkout the latest version of everything (including itself).

I spent the past week wrestling with this monstrosity. Ended up successfully writing a Conan package for it that builds for Windows and Linux (there's one on Conan center, but it only supports Linux). I have 3 more platforms to go, but I think it'll be a better idea to just scrap everything and refactor into something more reasonable using CMake.

Instead of Breakpad, they also have a newer one called Crashpad, which is meant to improve reliability on Mac OS. Unfortunately, it depends on Chromium, so it won't work for my purposes.

...so all I'm saying is, maybe don't use Google as a role model for your project infrastructure.

/end rant

Crashpad doesn't depend on Chromium, in fact it uses mini_chromium (a mostly copy pasta of base) to avoid having that large dep:

https://chromium.googlesource.com/chromium/mini_chromium/

https://chromium.googlesource.com/crashpad/crashpad/+/refs/h...

What's the issue you're having with Crashpad? Indeed the breakpad project is a mess by modern standards.

To be honest, I haven't looked into Crashpad that much because (from a quick glance) it seems like it requires a secondary process for crash handling, whereas with Breakpad you can do it in the same process. That, and the chromium thing is what turned me away from it. I guess I should probably look into it a bit more, especially the size of mini_chromium to see if it's reasonable for my needs before I go forking Breakpad.

In addition to the other sibling comment, crashpad is built with GN, which is basically Bazel and overall much easier to reason about than GYP.

I've come to really disagree with this approach. It's extremely hard to undo an entrenched system. My org has a gnarly monolith that works well enough that we can't justify spending any money refactoring it as it enters it's 12th year of service.

"There is nothing more permanent than a temporary solution"

I worked at a company that went through this transition from thousands of microrepos to a monorepo. The main driver was application security. When everybody did their own thing in their own repo, rolling a new version of a library with a critical security fix was very expensive. It was impossible to complete in a reasonable amount of time, because you'd frequently find legacy services that were on some previous major version that required rewriting code or upgrading other libraries to implement the fix.

A monorepo means every service is on the same set of versions of every dependency at a given point in time. It's much easier to reason about what is fixed, and move everything to a known good version.

More fancy words on somebody's CVs?

I worked at a similarly large and almost as popular SF based company and I can assure you every innovation there was 100% motivated by somebody's desie to put.it in their CV. Nothing of what was made there made any sense for the business, it was just people playing with toys. And worst part was people introducing this stuff (such as the one pushing for Lerna and an "UI" monorepo would leave the company after 1 year leaving a mountain of tech debt behind. But hey, they had Lerna in their CV now.

In some ways do you blame anyone? The industry has largely not optimized for retention by and large

The article clearly stated that they can commit a cross-cutting changes (a change that can affects thousands of binaries downstream) within 15 minutes.

I haven't done any calculation on how long that kind of changes can be done with polyrepo setup.

If they can do that, their story can only be better for single isolated changes. It seems from outside, Uber's monorepo development experience vastly better than many small startups with variety of development methodology (no matter monorepo, polyrepo, or continuous deployment, or waterfall development, or GitFlow (a type of waterfall)).

> so I wonder what's the advantage they get in using a monorepo.

Much easier to apply horizontal and vertical changes which extend outside individual services e.g. update the interface of a service, or fix a bad pattern.