News for hackers. Stuff that matters.
Re:We Halved Go Monorepo CI Build Time by masklinn on Sunday June 26, 2022 @08:09AM (#31883226)
> Alternatively don't use monorepos

And now you get into the fun situation of having to handle ecosystemic asynchronous updates between your multiple repositories.

Re:We Halved Go Monorepo CI Build Time by viraptor on Sunday June 26, 2022 @08:26AM (#31883310)
These are orthogonal concepts. If you don't have a monorepo, your CI can still build all downstream packages and redeploy changes on every update. If you have a monorepo, you can also ignore dependency changes and keep various services behind and in a broken-behaviour state if they'd get redeployed from main. Monorepo makes the sync updates slightly easier, but it's neither necessary nor forces it.
Re:We Halved Go Monorepo CI Build Time by dymk on Sunday June 26, 2022 @10:05AM (#31884122)
One commit in a monorepo now becomes N commits for every repo in every other repo
Re:We Halved Go Monorepo CI Build Time by morelisp on Sunday June 26, 2022 @11:46AM (#31885011)
Only if you need to update all N other projects every time, which is unusual.

Our services aren't even particularly well-factored and we update median 2, mode 1, mean 3-4 when we change a core library. (Out of ~25.)

Re:We Halved Go Monorepo CI Build Time by bobsomers on Sunday June 26, 2022 @02:00PM (#31886238)
> Only if you need to update all N other projects every time, which is unusual.

This isn't true in my experience at large companies. For this to be true, you would either need to have absolutely no shared core libraries, or core libraries that are so stable and unchanging that they never receive updates. Updates to those core libraries usually result in a cascade of changes across a ton of repos, tons of tests that need to be rerun, and a slew of now incompatible versions.

All of these problems go away with a monorepo. Monorepos are absolutely the right solution for 98% of companies out there because they will never reach the engineering scale where monorepos start to struggle. For the 2% of companies that do reach that engineering scale, these kinds of blog posts get written.

The mistake is reading blog posts like this and thinking is necessary or even relevant to any more than 2% of the companies out there. The vast majority of shops will never need anything like this.

Re:We Halved Go Monorepo CI Build Time by morelisp on Sunday June 26, 2022 @02:54PM (#31886777)
You seem to have ruled out the obvious choice of "downstream projects just don't update." We update the projects we need the new library behavior in. There's no reason to touch the other 20 until there's some new behavior they do need.

I don't really see this as a mono vs. multi repo issue either. In a mono repo you can still choose which projects update and the more you choose the bigger the diff and more involved the review process. In a multi repo case you can build analogous tooling to generate downstream MRs automatically.

If you need to touch every downstream project regularly just to keep things functional, you don't have a mono-repo, you have a monolith. (Which can also be fine, really! But it's not the same thing at all.)

(Yes, I know with 25 projects we're nowhere near Uber scale. But based on my experience at larger companies, this seems to scale more or less linearly - in terms of downstream project count, library count, and developer time available to dedicate to such things. If anything, developer time available is what seems to scale super-linearly.)

Re:We Halved Go Monorepo CI Build Time by GeneralMayhem on Sunday June 26, 2022 @06:54PM (#31888889)
The "downstream projects just don't update" option is one I'd consider completely unacceptable. What if the update is a security patch? What happens when they inevitably do want a new feature, and have to fast-forward through months or years of interface updates all at once? What happens when a developer wants to work on a feature that cuts across systems, and now has to re-learn the "old" way of doing things?
Re:We Halved Go Monorepo CI Build Time by morelisp on Monday June 27, 2022 @10:41AM (#31895224)
> What if the update is a security patch?

Even a security fix is unlikely to affect every downstream user of a library except in egregious cases. But, if so, yes you update them all. (I'd say this is much less frequent than, I don't know, OpenSSL or log4j having a bug that makes us do this. The specific concern of an internal library having such a broad vulnerability is negligible in influencing our CI design.)

> What happens when they inevitably do want a new feature, and have to fast-forward through months or years of interface updates all at once?

You do it.

> What happens when a developer wants to work on a feature that cuts across systems, and now has to re-learn the "old" way of doing things?

You do it.

I didn't say "it has no associated costs." But you need to weigh those costs against the other costs of a monorepo, and other costs in your CI/CD design generally. "Take longer to update a really old project once a year" is a much lower cost for us than "have a dedicated CI team to wrangle the tooling we need for automatic downstream pushes / a monorepo."

Re:We Halved Go Monorepo CI Build Time by viraptor on Sunday June 26, 2022 @09:19PM (#31889649)
Did you miss the "every time" qualifier? Everyone will have some core libraries, but most work will be very limited in scope. Outside of large companies you can see this effect in nixpkgs for example. See how few entries have a rebuild count larger than 10 https://github.com/NixOS/nixpkgs/pulls?q=is%3Apr+is%3Aclosed
Re:We Halved Go Monorepo CI Build Time by jogjayr on Sunday June 26, 2022 @05:21PM (#31888109)
> which is unusual

It's unusual because a multi-repo environment makes it difficult to do, not because it's not necessary.

Re:We Halved Go Monorepo CI Build Time by morelisp on Sunday June 26, 2022 @06:31PM (#31888748)
It’s not necessary. As proof: we don’t do it, yet our services run.
Re:We Halved Go Monorepo CI Build Time by elteto on Sunday June 26, 2022 @11:50AM (#31885043)
I never understood the hand-waiving around multirepos. Parallel, distributed updates are extremely hard to get right in software, but, for some magical reason, not when checking in code across N separate repos with interdependencies.
Re:We Halved Go Monorepo CI Build Time by pjmlp on Sunday June 26, 2022 @10:52AM (#31884533)
A much better scenario from my experience, given that not everyone is on the last version anyway.
Re:We Halved Go Monorepo CI Build Time by spion on Sunday June 26, 2022 @08:28AM (#31883320)
We know how to do that. Versioning, deprecations etc. We do that with all external dependencies.
Re:We Halved Go Monorepo CI Build Time by masklinn on Sunday June 26, 2022 @09:00AM (#31883505)
All of which adds more effort, more synchronisation, more delay, more processes, and more ceremony.

> We do that with all external dependencies.

Indeed. Why you’d want to also suffer that for internal dependencies when there is no reason to, I can’t fathom.

Re:We Halved Go Monorepo CI Build Time by spion on Sunday June 26, 2022 @09:42AM (#31883872)
I think this article clearly shows that the effort of handling a monorepo is way higher.

If you have a good process for external dependencies you can apply that to internal. If you don't have a good process for external dependencies a monorepo won't help (unless you make all your dependencies internal, which is even more expensive)

Re:We Halved Go Monorepo CI Build Time by rfoo on Sunday June 26, 2022 @12:29PM (#31885358)
> If you have a good process for external dependencies you can apply that to internal.

Yeah... but usually good process for external dependencies requires a lot of ceremony and takes maybe days to update one dependency. If the same happens to internal dependencies we will be just slower at making changes for no good reason.

This is more of a cultural thing. In fact, we are creating complicated solutions to mitigate this problem, only to go full circle after a while: people implemented Service Mesh via proxy sidecars, it added more layers (than just embedding this into the RPC library), but even doing so is easier than persuading downstream project owners update their dependency in a timely manner, so it takes off. And now they are going to chase for "proxyless" due to "performance". Oh well.

Re:We Halved Go Monorepo CI Build Time by spion on Sunday June 26, 2022 @12:44PM (#31885498)
Why would a complex dependency update be faster in a monorepo? In my experience the biggest savings are for simple updates that happen acorss multiple repos - when the update itself is complex, the extra time to make multiple PRs is dwarfed by the actual update's complexity, and it helps if the update can be separated into several projects where the version can be bumped separately.
Re:We Halved Go Monorepo CI Build Time by sagarm on Thursday June 30, 2022 @02:53AM (#31929393)
You don't "update" the dependency. You are always building and releasing at head with all your internal deps (external deps are still versioned of course).

> the extra time to make multiple PRs is dwarfed by the actual update's complexity

Writing code is never the majority of the work. All the operational stuff is more complex without forced upgrades: coordinating the dependency updates, automating integration testing, pushing clients to migrate off old thick clients, determining what version of a specific repo is deployed, running a bisect to root cause issues, etc.

Re:We Halved Go Monorepo CI Build Time by rfoo on Sunday June 26, 2022 @01:26PM (#31885901)
A lot of things could be done in a series of "simple updates that happen across multiple repos". Different style of doing the job I guess.

Also, having a rather global view of the impact of my code before actually make the change helps a lot, especially if you are working on low-ish level libraries. Technically this is not bound to monorepo, but people usually criticize the complexity introduced by having such ability for monorepo, so :)

Re:We Halved Go Monorepo CI Build Time by tynpeddler on Sunday June 26, 2022 @04:12PM (#31887500)
The process for rolling out breaking API changes is the same for monorepos as it is for multi-repos since, during a deploy, multiple versions of each service will be running simultaneously. The only advantage of a mono-repo is the atomic commit across multiple services. It's definitely possible through a combination of convention and tooling to do something similar with a multi-repo, but as of yet this is a less explored paradigm.
Re:We Halved Go Monorepo CI Build Time by oceanplexian on Sunday June 26, 2022 @01:14PM (#31885767)
As a Systems Engineer / SRE you’re damn right it adds more effort and delay. And it’s not because there’s “no reason” but because when you’re operating a production service at scale you need to put in the effort to make sure dependency changes won’t cause a massive outage or lose customer data. Automated tests and CI/CD only go so far and are not a cure-all.