'A well-designed system should not have X therefore you don't need Y.'
When 'A well-designed system should not have X' is just a matter of opinion, or doesn't acknowledge trade-offs that might make X the better option than Z, then this isn't a useful argument against Y.
at the risk of coming up with a contrived example: let's say you own a service that need to deserialize a datetime in a request in a format you don't currently support. assuming you own the stack, you need to a) update your date library b) update your webserver stack c) possibly update an intermediate webserver stack that includes primitives like logging, telemetry, tracing, auth, service discovery and d) your actual service.
If a->d are all independent, separate components, you have to orchestrate those changes through 4 separate repositories. And god forbid something you did at the lowest point in the stack is completely unworkable higher up.
There's all sorts of rocket science you could do to orchestrate these changes, but it ends up being contrived and edgecasey.
Most of the pain from monorepos can also be addressed with a dash of rocketscience (see:bazel), but the end model tends to have
a) have an easier mental model for the user
b) allow for consolidation of infrastructure work. Ie, your build/ci/language tooling teams can focus their efforts on one place
c) can coordinate changes across the entire stack within one field of view
d) can coral some of the worst, disparate instincts of a growing engineering org (ie, tons of teams optimizing for local maximas without internalizing knock-on effects).
e) fewer weird edgecases.
I like having examples, even contrived ones, but I'm not sure I understood this one. Can you elaborate on what you mean? Is it about adding support for a new serialization format for dates in requests to a service? Why would this affect the webserver stack and logging/telemetry/tracing/auth primitives?
I find that a lot of organizations have really strange thoughts on how to factor things into separate microservices and libraries. Usually I approach by asking the following question: if this was an open source-library or service (e.g. like elasticsearch), would you use it? If not, then its probably not a great candidate for a separate thing - lets try and come up with something else.
One way to handle CI/CD is using standardized pipelines e.g. you tag your repo with a tag `app:node` or `lib:js` and the github org pipeline scanner will find it and assign the standard `app:node` or `lib:js` pipeline to it.
A way that I like better but most tools unfortunatley don't support it yet is for the infra teams to publish libraries that are essentially functions taking some parameters and generating (standard) pipelines/configuration. Those can then be tracked together the same as other dependencies.
Besides the "N pull requests" problem you now lose history whenever you move a file across repo boundaries, and you'll eventually have straggler projects staying on old versions for years - so switching to a new way of doing something essentially means supporting both versions forever. Code for common needs gets duplicated, or worse, split out into yet another repository and /then/ duplicated, because no one wants to check if any of 100 repositories rely on the buggy behaviour they want to fix.
I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/. I'm surprised to see so many comments summarily dismissing them, as if sanely managing thousands of smaller interdependent repositories doesn't require at least as much investment in custom tooling.
The project should stay a full monolith until the factoring is more clear.
> I find this to be this a level headed explanation of the advantages of monorepos: https://danluu.com/monorepo/.
There are two issues I have with this article. One of them is when it describes drawbacks of multiple repositories, its not specific enough e.g.
> That sounds like it ought to be straightforward, but in practice, most solutions are cumbersome and involve a lot of overhead.
The other is that it assumes you have Google scale of resources to throw at the problem. If thats the case you can make anything work / monorepos or polyrepos. The issue is that small-to-medium sized organizations will not be prepared to invest the amount of resources needed to keep a monorepo working well, as the org will largly need to rely on existing available (OSS) tools which often have poor monorepo support. (Bazelifying everything has a significant cost, bazel rules are often not generic enough to work with e.g. the variety of JS ecosystem tools)
If one business change requires code changes to ~every module in your source tree, then of course microservices, and therefore separate repos per service or whatever, make no sense at all.
Sure, each project team could upgrade independently, but the company that’s chosen a mono-repo for their go code is likely to desire to have a single team tackle this upgrade.
In practice, you often end up with larger changes which are mostly janitorial. Bumping a version of a log library across the infrastructure, or a version of a serialization system. This simplifies dependency convergence and a best handled as if they are cross-project changes.
The other problem is that over time, good factoring tend to deteriorate. A large software project will invariably have people with different brains working on it, and they'll have different needs for the factoring. So the project naturally pushes itself toward situations where cross-project changes become a necessary thing.
Good factoring can probably be quantified too: dependency chains should be shallow and graph connectivity should be low. The more dependants a module has, the more stable its API contract should be.
i’m finding it hard to think of a system at a large company that would have no internal libraries used by multiple projects, that wouldn’t require ever making a cross project change.
What helps maintain polyrepos sane
1. A healthy methodology for dependency management
Evolve APIs with deprecations. Decide on a healthy amount of time for a deprecations to live. Set up alerts when deprecations reach certain amount of time. Set up dashboards (e.g. Grafana) to track dependencies. Help other teams update, and think about how to make updates less painful.
2. Define good boundaries and API contracts to adhere to. The most important thing for a good API contract is stability.
3. Don't prematurely split into microservices. Better ideas for stable API contracts emerge the longer you can wait.
1. I don't see how this is a problem specific to polyrepos. Have an "open PRs" link in the onboarding handbook that gives you a view of pull requests from all repos in the organization. GitHub automatically shows you notifications from all repos. If engineers still chose to focus on one or two repos after that, I'm not sure why.
- Have a (Grafana) dashboard where you can see the latest / newest stuff. Use standard GH tools you use for OSS, such as follows etc to keep up.
2. Don't prematurely split into multiple repos. "No monorepo" doesn't mean not having poly-package repos. It means thinking what the sensible (library or service) API boundary is - treating your projects as you would treat library / service development. In this case a separate repo with lib3, lib2 and lib1 sounds like a good way to go - at most one repo per orthogonal internal framework (e.g. core-react-components). Repo dependency chains should be as shallow as possible, and differenting between public and internal packages is important.
3. Help other teams upgrade. If you are responsible for repo A, once you publish a new version tagged appropriately with semver, use the dashboard to look at your dependants and work with them (or rather, for them) to upgrade. Think of your dependants as internal customers, and make sure you add enough value for them to justify the upgrade effort. Cultivate a culture that values updates.
4. There are other alternatives to `npm link` e.g. see `yalc` https://github.com/wclr/yalc
Another pet peeve of mine is that the real issues get lost when you try to generalize. The article attempts to do this but that makes it hard to evaluate its claims. The best way to evaluate (alternative) solutions is to take a more concrete example repo.
For scalability of this model I'll just point to the OSS community; individual maintainers often several dozen active repositories, but also they have an API contract worthy of a documentation website, versioning scheme and planned deprecation, and they typically avoid cross-project dependencies
No, that way lies sorrow and despair. There should only be one version of any dependency being used in your company. Any deviations should require like VP-level approvals or something.