First, VCS ACLs will massively reduce the benefits you're supposed to get from a monorepo. How will you do global refactors in that kind of a situation? How does a maintainer of a library figure out how the clients are actually using it? (The clients must have visibility into the library, but the opposite it unlikely to be true.)
Second, let's say that I maintain a library with a supported public interface that's implemented in terms of an internal interface that nobody's supposed to use. How will VCS ACLs allow me to hide the implementation but not the interface? When they kick off a build, the compiler needs to be able to read the implementation parts to actually build the library. It can't be that the clients have access to read the headers but then link against a pre-build binary blob. At that point you don't have a monorepo, you've got multirepos stored in a monorepo.
The actual solution are build system ACLs. Not ACLs for people, but ACLs for projects. Anyone can read the code, but you can say "only source files in directory X can include this header" or "only build files in directory Y can link against this object file".
> How will VCS ACLs allow me to hide the implementation but not the interface?
If you don't give people access to the code, they can't build it. So what? Publish pre-built binaries from your CI system back to source control.
> At that point you don't have a monorepo, you've got multirepos stored in a monorepo.
I think it's a spectrum. It would be stupid to dogmatically stick to either extreme. You modify things in a pragmatic fashion to solve the problems you're facing. In my experience, starting with a monorepo and making exceptions as needed has worked better than the alternative.
Your post sounds similar to a lot of the multi/mono repo discussions. You've focused on one problem and one way to solve that problem without considering that there are many ways to work around it. Neither approach is going to be pain-free and both require tooling for special scenarios.
I agree
> The actual solution are build system ACLs.
Or, maybe, better languages enforcing better design. In most of the cases artifacts and libraries are not related to the domain, engineers create them just to establish artificial boundaries between code components, isolate irrelated things, enforce encapsulation and avoid accidental mixing of metalanguages.
It would be lot better to have a smart compiler for this.
A tool which can prevent us from mixing different abstraction layers, creating unneccessary horizontal links between our components, etc, etc.
I have a couple of ideas how such a thing may look like.
It's a blogpost and the author didn't try to build a total and exhaustive formal system. These shortcomings are not absolute truth but actually they are true.
I've seen this multiple times: a small projects evolves over years into a monster. Engineers add new components and reuse any other components they may need creating horizontal links. At some point they feel like they lost their productivity and they blame monorepo because it's easy to create horizontal links in a typical monorepo. So, they try to build a multirepo flow and they spend a lot of effort, time and money trying to make it working. At some point they feel that their productivity is even worse than it was before because now they need to orchestrate things so they merge everything back.
Same applies not only to VCS flows, but to system design as well.
When we discuss monolith/microservices controversy all the monorepo/multirepo arguments may be isomorphically translated to that domain. What is better, monolithic app or a bunch of microservices? A role-based app of course: https://github.com/7mind/slides/blob/master/02-roles/target/...
Monorepo advocates are typically advocating for microservices, but within a single code base.
The way you provide access control is through code review and build system visibility.
In order to modify another group's code you require their approval on the review for that section of the code base. (Using mechanisms like github/gitlab owners files or rules within upsource.)
This still means that if one group needs to make extensive changes to another groups code, the path of least resistance may be to fork it into your own group's section of the repo.
Build tools provide another point of control. If you're using a tool like bazel, the way you link to a component in another portion of the repo is through target names. The only targets your code will have access to are those that the owners has declared as being available for external builds.
Yes and no. In both the cases it's a story about components and their isolation.
> they may end up building a distributed monolith
Yup, seen that many times.
> Monorepo advocates are typically advocating for microservices, but within a single code base.
I'm avocating roles. Everywhere.
> If you're using a tool like bazel
If only Bazel supports Scala well enough...
Many companies build their Scala code using Bazel[1]. For example, Databricks wrote about their experience using Bazel on a monorepo containing mostly Scala[2]. Can you share the specific concerns or issues you faced? Thanks.
[1] https://github.com/bazelbuild/rules_scala/blob/master/README... [2] https://databricks.com/blog/2019/02/27/speedy-scala-builds-w...
(Disclaimer: I work on Bazel)
Sure you could just use a manyrepo style of dependency tracking in a monorepo but I think that's not exactly what the author is exploring.
From what I read that is a correct assessment. What the OP is proposing is something of a strawman argument. No advocate of monorepos I've ever met believe that a monorepo should imply a monolith.
Generally they're advocating monorepos in order to develop microservices faster, and with less effort. Using a monorepo and the associated tooling side steps the pain that comes from complicated CI, the difficulty of sharing code, the difficulties of non-atomic cross-repo reviews, and the difficulties of making multi-app refactorings.
>Surely source control is for source code?
This is just pedantry. Checking in binaries is a pragmatic solution that solves a lot of problems.
Suppose the binaries are build byproducts, and people just check this stuff in, like, whatever. Well, if somebody needs to sign off on the output, that's a problem - so that person then doesn't use what's in the repo, but instead builds the output from scratch, from the source code, hopefully with known build tools (see above!), and signs off on whatever comes out.
But, day to day, for your average build, which is going to be run on your own PC and nowhere else, nobody need sign off on anything. If you link with some random object file that was built on a colleague's machine, say, then that's probably absolutely fine - and even if it isn't, it's still probably fine enough to be getting on with for now. If you work for the sort of company that's worried about this stuff, there's a QA department, so any issues arising are not going to get very far.
Overall, this stuff sorts itself out over time. Things that are problems end up having procedures introduced to ensure that they stop happening. And things that are non-problems just... continue to happen.
For simple things, if the code in a directory changes then the CI system does a rebuild of that directory. You can have the CI system either validate that the binary matches or commit the binary itself. More complicated things you'll have a build system such as Bazel which figures out what changed.
Depends if you want to wait for the CI system to upload or not. Also if you want CI to have commit permissions.
>Never mind that having a bit-for-bit reproducible build is incredibly difficult.
Debian is at something like 90% reproducible packages once they fix two outstanding things. Most languages will have settings and best practices at this point that will give reproducible builds.
>Anyway, such simple cases where a whole app lives under a single directory are vanishingly rare.
Then use Bezel once you get past that stage.
Look, to be blunt, it seems like you're trying to nitpick whatever anyone says while ignoring large parts of answers. Fact is, many people at small and large companies use monorepos successfully. They work for those people, you can keep trying to argue they don't or try to learn why they do.
I guess you could deploy first and verify automatically later. Hadn’t thought of that.
> Debian is at something like 90% reproducible packages once they fix two outstanding things. Most languages will have settings and best practices at this point that will give reproducible builds.
Never the less, getting (and keeping) bit-for-bit reproducibility is a ton of work, especially for software that changes every day, and the benefits aren’t compelling for many projects.
> Then use Bezel once you get past that stage.
This seems to be the answer, but it’s not very satisfying since Bazel’s support for many popular languages (e.g., Python) is lacking and there are lots of rough edges to iron out.
> Look, to be blunt, it seems like you're trying to nitpick whatever anyone says while ignoring large parts of answers. Fact is, many people at small and large companies use monorepos successfully. They work for those people, you can keep trying to argue they don't or try to learn why they do.
I never understand why people get defensive about things like this. I’m not attacking monorepos. I manage a monorepo at my small company, and I’ve run into lots of issues trying to make it work. I’m here trying to understand why so many people rave about monorepos, but often don’t have good answers for things like “how to manage rebuilds?”. You see this as “nitpicking”, but the distinction between “just git diff a directory!” and “use something like Bazel” is important.
This doesn't work well for dependencies where you're expected to be using the latest version of something that changes 10 times a day.
The rest of your questions are fairly irrelevant as they would be answered the same way as the in the dependency repo case. ie, use official binaries.
...but this is closer to multi-repo than monorepo. If you're in a monorepo you might as well use the source.
By the CI. All major CI/CD tools support rules like build binary x whenever a file under x-src/* changes; commit binary x when the ref matches /v[0-9.]+/; don't allow developers to manually push to these refs / paths; (run a script to) bump the dependent x of y whenever binary x changes; merge the bumped version if all tests still pass; etc.
A compiler is just a program that takes some input and create some output. Both the compiler and the input can have a cryptographically secure hash. Putting both in a sealed box, like a docker image, with its own hash, gives you a program that takes no input and produces some output.
If the box changes, run it in a trusted machine and save the output together with a signed declaration of which box version produced it
(See also: trusting trust)
Edit: I think we’re getting there though, with all the efforts going on with containers, webassembly blockchains, ipfs and so forth it’s getting closer
Usually somebody else has already gone through the work of doing it for you. Sometimes there are tools that do the translation for you. For example, Go modules are quite easy to translate to a BUILD file.
It’s actually not as bad as it sounds. You only have to do the hard stuff once, and every engineer in the org who uses it in the future is thankful for it.
Git's design can limit its usefulness in this respect - though perhaps you could solve this to some extent with git LFS? - but not all version control systems have this problem.
(partial clones avoid this, but, as git isn't designed for this use case, grabbing all of history happens far too easily.)
I feel like these debates are often fueled by false arguments. Either way you go, you're going to want to build support tools and processes to tailor your VCS to your local needs.