I can change a dependency and my code at the same time and not need to wait for the change to get picked up and deployed separately. (If they are in the same binary. Still need cross binary changes to be made in order and be rollback safe and all that.)

Have you ever worked at FB / Google / whatever other company has huge mono repo with great tooling?

I went from many years at FB, to a place like you describe - hundreds of small repos, all versioned. It’s a nightmare to change anything. Endless git cloning and pulling and rebase. Endless issues since every repo ends up being configured slightly differently, and very hard to keep the repo metadata (think stuff like commit rules, merge rules, etc) up to date. It’s seriously much harder to be productive than with a well-oiled monorepo.

With a monorepo, you wanna update some library code to slightly change its API? Great, put up a code change for it, and you’ll quickly see whether it’s compatible or not with the rest of the whole codebase, and you can then fix whatever build issues arise, and then be confident it works everywhere wherever it’s imported. It might sound fragile, but it really isn’t if the tooling is there.

How do you handle tagging, the whole repo gets a new tag / version every time something change?

Yes, all changes (CLs) are atomic operations. The CL number reflects a specific moment in time for the entire repo.

I have worked at a company that has huge monorepos and bad tooling.

Tooling isn't the problem though, the problems are:

- multiple monorepos copying code from each other, despite that code should be a library or installable python package or even deb package of its own

- you will never understand the entire monorepo, so you will never understand what things you might break. with polyrepos different parts can be locked down to different versions of other parts. imagine if every machine learning model had a copy of the pytorch source in it instead of just specifying torch==2.1.0 in requirements.txt?

- "dockerize the pile of mess and ship" which doesn't work well if your user wants to use it inside another container

- any time you want to commit code, 50000 people have committed code in-between and you're already behind on 10 refactors. by the time you refactor so that your change works, 4000 more commits have happened

- the monorepo takes 1 hour to compile, with nothing to compile and unit test only a part of it

- ownership of different parts of the codebase is difficult to track; code reviews are a mess

All of these are solved with tooling.

- dont have multiple monorepos

- use blaze or similar and run all downstream tests, the binary for your ml model includes pytorch whether you build from source or requirements.txt.

- other people committing doesn't matter if you are isolating presubmits and similar.

- using a blaze-like you never compile the whole monorepo

- code owners etc. makes this straightforward.

Like, as someone whose career has been mostly at Google, these are not problems I encounter at all, or only in cases where you'd have similar scope of problems no matter the repo structure.

> other people committing doesn't matter if you are isolating presubmits and similar.

Could you elaborate on this one?

If you only need to run the tests you affect, and only need to sync and update files touched in your CL, external changes are generally not impactful, sync & submit is a quick process even if people are submitting things elsewhere.

It's only a problem if someone submits a file you're touching, in which case you just have standard merge conflict issues.

I think all of your problems are root caused by the phrase "multiple monorepos". This sounds more like "polyrepos" or something, multiple siloed repos that themselves might contained disparate projects, languages and tooling.

Google3 is a true monorepo. 99% of code in the company is one repo, with minor exceptions for certain open source projects and locked down code.

Edit: for example, you can change YouTube code, Google Assistant code, a Maps API, some config for how URLs are routed at top level load balancers, etc all in one CL if you really wanted to/needed to.

Up to what scale?

This works well for a couple dozen repos per team in my experience. It’s also my preferred way to work.

It doesn’t scale so well to hundreds of repos per team without significant tooling. At some point anything cross-cutting (build tool updates, library updates, etc) becomes hard to track. Repos are left behind as folks change teams and teams are reorg’d.

I’ve never worked in a monorepo, but I can see the appeal for large, atomic changes especially.

If you have a reusable component in a separate repository and need a change, you have to submit that, merge, release, then bump the version in the downstream project to use it. Then if someone else uses the project updates but hits an issue that you introduced they have to go fix it, perhaps a month later, with no context of what changed. Or they just don’t upgrade the version, and reimplement what they need. With a monorepo it would be one change, and your change breaking someone else’s would get flagged and fixed with the code change. I’ve seen the amount of shared code get less and less and more stale with polyrepo.