And that way, you can't have atomic updates across the repositories and need to synchronize them all the time, great.

We have repository systems built for centralized atomic updates, and giant monorepos, like SVN. Question is why are we trying to have Git do this, which was explicitly designed with the exact opposite goal? Is this an attempt to do SVN in Git so we get to keep the benefits of the former, and the cool buzzword-factor of the latter? I don't know.

Also when I try to think about reasons to have atomic cross-project changes, my mind keeps drawing negative examples, such as another team changing the code on your project, is that a good practice? Not really. Well unless all projects are owned by the same team, it'll happen in a monorepo.

Atomic updates not scaling beyond certain technical level is often a good thing, because they also don't scale on human and organizational level.

1. You determine that a library used by a sizable fraction of the code in your entire org has a problem that’s critical to fix (maybe a security issue, or maybe the change could just save millions of dollars in compute resources, etc.), but the fix requires updating the use of that library in ~30 call sites spread across the codebases of ~10 different teams.

2. You create a PR that fixes the code and the problematic call sites in a single commit. It gets merged and you’re done.

In the multi-repo world, you need to instead:

1. Add conditional branching in your library so that it supports both the old behavior and new behavior. This could be an experiment flag, a new method DoSomethingV2, a new constructor arg, etc. Depending on how you do this, you might dramatically increase the number of call sites that need to be modified.

2. Either wait for all the problematic clients to update to the new version of your library, or create PRs to manually bump their version. Whoops - turns out a couple of them were on a very old version, and the upgrade is non-trivial. Now that’s your problem to resolve before you proceed.

3. Create PRs to modify the calling code in every repo that includes problematic calls, and follow up with 10 different reviewers to get them merged.

4. If you still have the stamina, go through steps 1-3 again to clean up the conditional logic you added to your library in step 1.

Basically, if code calls libraries that exist in different repos, then making backwards-incompatible changes to those libraries becomes extremely expensive. This is bad, because sometimes backwards-incompatible changes would have very high value.

If the numbers from my example were higher (e.g. 1000 call sites across 100 teams), then the library maintainer in a monorepo would probably still want to use a feature flag or similar to avoid trying to merge a commit that affects 1000 files in one go. However, the library maintainer’s job is still dramatically easier, because they don’t have to deal with 100 individual repos, and they don’t need to do anything to ensure that everyone is using the latest version of their library.

Your monorepo scenario makes the following unlikely assumptions:

1. A critical security/performance fix has no other recourse than breaking the interface compatibility of a library. Far more common scenario is this can be fixed in the implementation without BC breaks (otherwise systems like semver wouldn't make sense).

2. The person maintaining the library knows the codebases of 10 teams better than the those 10 teams, so that person can patch their projects better and faster than the actual teams.

As a library maintainer, you know the interface of your library. But that's merely the "how" on the other end of those 30 call sites. You don't know the "why". You can easily break their projects, despite your code compiles just fine. So that'd be reckless of an approach.

Also your multi-repo scenario is artificially contrived. No, you don't need conditional branching and all this nonsense.

In the common scenario, you just push a patch that maintains BC and tell the teams to update and that's it.

And if you do have BC breaks, then:

1. Push a major version with the BC breaks and the fix.

2. Push a patch version deprecating that release and telling developers to update.

That's it. You don't need all this nonsense you listed.

I’ve lived both lives. It absolutely is an ordeal making changes across repos. The model you are highlighting opens up substantial risk that folks don’t update in a timely manner. What you are describing is basically just throwing code over the wall and hoping for the best.

We're still people capable of speech and communication, if something is that urgent, we can communicate it to our peers.

Changing code under their nose risks breaking bunch of projects. We can also fix this by rather communicating, right? But if we CAN communicate... then we can go back to the previous option (telling them to update) as it becomes just as viable.

Communicating is always essential, and can't be avoided.

Semver is a second-rate coping mechanism for when better coordination systems don't exist.

Patching the code of 10 projects you don't maintain isn't an example of a "coordination system". It's an example of avoiding having one.

In multithreading this would be basically mutable shared state with no coordination. Every thread sees everything, and is free to mutate any of it at any point. Which as we all know is a best practice in multithreading /s

The same code can have multiple overlapping sets of maintainers. For example, one team can be responsible for business logic while another team can manage core abstractions shared by many product teams. Yet another team may be responsible for upgrading to newer toolchains and language features. They'll all want to touch the same code but make different, roughly orthogonal changes to it.

Semver provides just a few bits of information, not nearly enough to cover the whole gamut of shared and distributed responsibility.

The comparison with multithreading is not really valid, since monorepos typically linearize history.

Semver was enough for me to resolve very simply a scenario above that was presented as some kind of unsurmountable nightmare. So I think Semver is just fine. It's an example of a simple, well designed abstraction. Having "more bits" is not a virtue here.

I could have some comments on your "overlapping responsibilities" as well, but your description is too abstract and vague to address, so I'm pass on that. But you literally described the concept of library at one point. There's nothing overlapping about it.

> You create a PR that fixes the code and the problematic call sites in a single commit. It gets merged and you’re done.

What happens when you roll this out and partway through the rollout an old version talks to a new version? I thought you still needed backwards compat? I'm a student and I've never worked on a project with no-downtime deploys, so I'm interested in how this can be possible.

Version skew is only an issue when there's cross service communication. One service deployed on two different codepaths (in this case, using a different library implementation) is completely fine.

My example was a library, so both the library and it’s caller are in the same release artifact (binary/JAR/whatever).

If you’re changing the interface of an RPC service, then you can’t do that in a single commit, and need to fall back to something like the second approach, but with even more caution to make sure you properly account for releases and the possibility of rollbacks.

The philosophy differences between git and svn aren't limited to one centralized repo vs. multiple repos.

With git, you get a local stage/history which lets you rework/reorder your commits for clarity before pushing. It also allows for more options to resolve conflicts, although this increased ability has brought its own problems.

I also find git’s branching model to be far superior to subversion’s.

I assume that this is due to its design center around distributed repositories and patch files. But it’s still there even when using a central repo and a monorepo structure.

Of course I want people who care about modernizing code to come in and modernize my code (such as upgrades to newer language versions). Why should the burden be distributed when it can be concentrated among experts?

I leverage type systems and write tests to catch any mistakes they might make.

What do atomic source updates get you if you don't have atomic deploys? I'm just a student but my impression is that literally no one serious has atomic deploys, not even Google, because the only way to do it is scheduled downtime.

If you need to handle different versions talking to each other in production it doesn't seem any harder to also deal with different versions in source, and I'd worry atomic updates to source would give a false sense of security in deployment.

It's not just atomic updates, the critical part is to run all the relevant integration jobs. The aim is that when a change goes to the reference branch, you must understand what other code is going to get impacted. Without that, you can break builds depending on the repo you changed, and if the broken repos don't change often, the breakage could be discovered months later. If you're in that situation, you'll randomly discover piles of debt when you use a slow-moving repo.

Atomic deploys are not as important, because you can still decide to version your APIs or releases even if you're using a monorepo.

That being said, you can use multiple repos and still mostly avoid trouble by choosing how to cut your codebase (HR software is likely not going to depend heavily on presale, for instance). The metric to optimize is to minimize the required version bumps.

Got has submodules which solves this but for some reason people decided to stop using them a while back.

Yes, for good reason: they're a poor solution to most needs, are needlessly error prone, and so they make life more difficult especially among components with heavy developer traffic. As someone who's used both, I vastly prefer the monorepo approach, even if git is kind of slow sometimes.

> If you need to handle different versions talking to each other in production it doesn't seem any harder to also deal with different versions in source

It's much more annoying to deal with multi-repo setups and it can be a real productivity killer. Additionally, if you have a shared dependency, now you have to juggle managing that shared dep. For example, repo A needs shared lib [email protected] and repo B needs [email protected], because developers on team A didn't update their dependencies often enough to keep up with version bumps from the Foo team. Now there's a really weird situation going on in your company where not all teams are on the same page. A naiive monorepo forces that shared dep change to be applied across the board at once.

Edit: In regards to your "old code talking to new version" problem, that's a culture problem IMO. At work we must always consider the fact that a deployment rollout takes time, so our changes in sensitive areas (controllers, jobs, etc) should be as backwards compatible as possible for that one deploy barring a rollback of some kind. We have linting rules and a very stupid bot that posts a message reminding us of that fact if we're trying to change something sensitive to version changes, but the main thing that keeps it all sane is we have it all collectively drilled in our heads from the first time that we deploy to production that we support N number of versions backwards. Since we're in a monorepo, the PR to rip out the backwards compat check is usually ripped out immediately after a deployment is verified as good. In a multi-repo setup, ripping that compat check out would require _another_ version bump and N number of PRs to make sure that everyone is on the same page. It really sucks.

Yes you can, it happens when you bump the sub module reference. This is how reasonable people use git.

Submodules often provide a terrible user experience because they are locked to a single version. To propagate a single commit, you need to update every single dependent repository. In some contexts that can be helpful, but in my experience it's mostly an enormous hassle.

Also it's awful that a simple git pull doesn't actually pull updated submodules, you need to run git submodule update (or sync or whatever it is) as well.

I don't want to work with git submodules ever again. The idea is nice, but the user experience is really terrible.

And woe unto junior developers who change into the submodule directory and do a git commit, then made infinitely worse if it's followed by git push because now there's a sha hanging out in the repo which works on one machine but that no one else's submodule update will see without surgery

I'm not at my computer to see if modern git prohibits that behavior, but it is indicative of the "watch out" that comes with advanced git usage: it is a very sharp knife

Looking back I just do not understand why git came up with this awkward mess of submodules. Instead it should have a way to say that a particular directory is self-contained and any commit affecting it should be two objects. The first is the commit object for the directory using only relative paths. The second is commit for the rest of code with a reference to it. Then one can just pull any repository into the main repository without and use it normally.

git subtree tries to emulate that, but it does not scale to huge repositories as it needs to change all commits in the subtree to use new nested paths.

Or define your interfaces properly, version them, and publish libraries (precompiled, ideally) somewhere outside of your source repo. Your associated projects depend on those rather than random chunks of code that happen to be in the same file structure. It's more work, but it encourages better organization in general and saves an incredible amount of time later on for any complex project.

I don’t like this because it assumes that all of those repositories are accessible all of the time to everyone who might want to build something. If one repo for some core artifact becomes unreachable, everyone is dead in the water.

Ideally “cached on the network” could be a sort of optional side effect, like with Nix, but you can still reproducibly build from source. That said, I can’t recommend Nix, not for philosophical reasons, but for lots of implementation details.

Using the expression "that's how reasonable people do ..." is not a great conversation starter.

I've always had a bad experience using submodules, they're imo the poor developer's versioning tool. It's useful when you use a language without a good build/packaging tool, but otherwise, I'm better off leaving the language-specific tool fetch the depended code.

If the project has good separation of concerns, you don't need atomic updates. Good separation of concerns yields many benefits beyond ease of project management. It requires a bit more thought, but if done correctly, it's worth many times the effort.

Good separation of concerns is like earning compound interest on your code.

Just keep the dependencies generic and tailor the higher level logic to the business domain. Then you rarely need to update the dependencies.

I've been doing this on commercial projects (to much success) for decades; before most of the down-voters on here even wrote their first hello world programs.