Working in environments where different people have partial access to different parts of the code never felt productive to me -- often, the time to figure out who can take on a task and how to grant all the access might take longer than the task itself.
The reason git doesn't have partial repo cloning is because it was written by people without regard to past experience of software development organizations. It is suited to the radically decentralized group of Linux maintainers. It is likely that your organization much more closely resembles Google or Facebook than Linux. Perforce has had partial checkout since ~always, because that's a pretty obvious requirement when you stop and think about what software development companies do all day.
I’ve been using Git/Hg for years and I still run into the occasional Gitastrophe where I have to Google how to unbreak myself.
From my experience git/vcs is not an issue for monorepo. Build, test, automation, deployments, CI/CD are way harder. You will end with a bunch of shell scripts, make files, grunt and a combination of ugly hacks. If you are smart you will adopt something like bazel and have a dedicated tooling team. If you see everything as nail, you will split monorepo into an unmaintainable mess of small repos that slowly rot away.
This means that every single commit every single test suite has to be run across the whole organization. For a small startup that's probably fine, but for a larger organization that adds a ton of issues and extra time. If your change to a shared project breaks another project that you may not be familiar with you end up not only with a delayed context switch (since you don't know about it until the CI has finished running all tests across the whole org), but now you have to figure out how to address it before someone else changes code in that shared project which causes a merge conflict.
Compare that to a split-repo setup where projects are sharing code indirectly (via package management systems). When you change code other projects will only consume the new code when they actually upgrade their packages, and usually a developer that is familiar with the code base will actively hit that breakage in real time.
Granted, this has its own trade offs such as all your systems may be running different versions of shared code, and the delay in breakage detection can cause some back and forth in breakage between isolated projects.
That’s my take anyway
That X is mostly a function of how open you are to the idea that maintaining + using a centralized, language-agnostic build graph is a Useful Thing.
But once that X is crossed, I don't think it's possible to happily go back to a non-monorepo environment. :)
FWIW, we've used Pants successfully for years, which has excellent Python support. However, we're in a long-haul migration to Bazel, so you might be better off revisiting Bazel if you want to invest in something that's extremely likely to issue dividends over the next 10 years.
You could probably use jsonnet expressions and invoke them from a pre-commit hook.
That seems fairly elegant to me.
Nix bills itself as a fully reproducible package manager and Bazel as a fully reproducible build tool. In the abstract, both of these reproducibly build software, so that puts them in the same category in general.
> Can you re-run Make in a pipeline to verify that the actions in use match what would be generated based on the repo state?
Yes, and I’m doing something similar although not with Make. The problem is that this home-grown build system doesn’t do incremental builds, so everything will be fully built from source every time, and that can take a while for very deep dependency graphs.
Conceivably your “Actions generator” script could generate only the jobs that need to be run based on what changed since the previous commit, referencing artifacts from the previous commit for those build steps which weren’t invalidated. This is a neat concept, but we’re well on our way to reinventing Nix or Bazel.
And I agree with the general point that monorepos require a great build tooling as a match.
Note: I don't quite like Bazel; but my take is that Bazel supports too much configurations and options: my workflows are always much simpler. Which is why I am curious to hear what do your workflows require.
Overall, custom toolchains are common (especially, in the embedded world), and a good build system should make that very easy.
I also really like monorepos, but Git and GitHub really don't work well at all for them.
On the Git side there is no way to clone only parts of a repo or to limit access by user. All the Git tooling out there, from CLI to the various IDE integrations, are all very ill adjusted to a huge repo with lots of unrelated commits.
On the Github side there is no separation between the different parts of a monorepo in the UI (issues, prs, CI), the workflows, or the permission system. Sure, you can hack something together with labels and custom bots, but it always feels like a hack.
Using Git(hub) for monorepos is really painful in my experience.
There is a reason why Google, Facebook et all have heaps of custom tooling.