- dont have multiple monorepos
- use blaze or similar and run all downstream tests, the binary for your ml model includes pytorch whether you build from source or requirements.txt.
- other people committing doesn't matter if you are isolating presubmits and similar.
- using a blaze-like you never compile the whole monorepo
- code owners etc. makes this straightforward.
Like, as someone whose career has been mostly at Google, these are not problems I encounter at all, or only in cases where you'd have similar scope of problems no matter the repo structure.
Could you elaborate on this one?
It's only a problem if someone submits a file you're touching, in which case you just have standard merge conflict issues.
Google3 is a true monorepo. 99% of code in the company is one repo, with minor exceptions for certain open source projects and locked down code.
Edit: for example, you can change YouTube code, Google Assistant code, a Maps API, some config for how URLs are routed at top level load balancers, etc all in one CL if you really wanted to/needed to.
Tooling isn't the problem though, the problems are:
- multiple monorepos copying code from each other, despite that code should be a library or installable python package or even deb package of its own
- you will never understand the entire monorepo, so you will never understand what things you might break. with polyrepos different parts can be locked down to different versions of other parts. imagine if every machine learning model had a copy of the pytorch source in it instead of just specifying torch==2.1.0 in requirements.txt?
- "dockerize the pile of mess and ship" which doesn't work well if your user wants to use it inside another container
- any time you want to commit code, 50000 people have committed code in-between and you're already behind on 10 refactors. by the time you refactor so that your change works, 4000 more commits have happened
- the monorepo takes 1 hour to compile, with nothing to compile and unit test only a part of it
- ownership of different parts of the codebase is difficult to track; code reviews are a mess