We went from monorepo to multi-repo at work and it's been a huge set back and disappointment with the devs because it's what our contractors recommended.
I've asked for a code deploy and everything and it's failed in prod due to a missing check in
> ...because it's what our contractors recommended
A startup I joined started with a multi-repo because the senior team came from a FAANG where this was common practice to have multiple services and a repo for each service.
Problem was that it was a startup with one team of 6 devs and each of the pieces was connected by REST APIs. So now any change to one service required deploying that service and pulling down the OpenAPI spec to regenerate client bindings. It was so clumsy and easy to make simple mistakes.
I refactored the whole thing in one weekend into a monorepo , collapsed the handful of services into one service, and we never looked back.
That refactoring and a later paper out of Google actually inspired me to write this article as a practical guide to building a "modular monolith": https://chrlschn.dev/blog/2024/01/a-practical-guide-to-modul...
The difference is that past a certain level of complexity, the org will most certainly need specialized tooling to support massive codebases to make CI/CD (build, test, deploy, etc.) times sane.
On the other hand, multi-repos may work for massive orgs, but is always going to add friction for small orgs.
Eeh... Might be quite meta-, but really doesn't feel very mono-.
The point here is to understand what are the problems that are being solved, understand if they are similar to yours, and make a decision based on wether the tradeoffs are a good fit for you.
Not necessarily disagreeing with you but I just feel the pendulum on this statement has swung to far to the other side now.
Yes it is. You said so yourself:
> The point here is to understand what are the problems that are being solved, understand if they are similar to yours, and make a decision based on wether the tradeoffs are a good fit for you.
That's you agreeing: “understand and make a decision” is pretty much the exact opposite of “adopt anything by default” — which peoples takeaway is that you shouldn't do — isn't it? So you're agreeing with the general takeaway, not arguing against it.
I think this pendulum has stopped swinging and come to a rest. And pendulums (pendula? penduli?) always stop pointing straight down, midway between the extremes: You shouldn't mindlessly reject something just because it comes from a big company, but certainly not blindly adopt it just because it does, either. But, just as you said, understand if it fits for you and then decide based on that.
And hey, on this particular issue, it seems some big companies do it one way, and others the opposite — so maybe the (non-existent) “average big company” is halfway between the extremes, too. (Whatever that means; one-and-a-half repository? Or half an infinity of repositories?)
If you just have 20-30 devs and everyone is pretty silo'd (e.g. frontend or backend, data or API, etc) having 75 repos for your stuff is just silly.
As someone who worked in the dev tooling org, getting teams to keep their deps up to date was a nightmare.
Then you are adapting your project to the properties of code repository. I don't see that as a benefit.
Were they working with multiple services in a multi-repo? Seems like a cross-product explosion of repos. Did that configuration inhibit releases, or was the process cumbersome but just smooth because it was so rote?
So yeah, cumbersome, but established, and huge downside risk to messing with the status quo. It was basically Git applied on top of an existing “copy the source” release process.
A github search for 'operator' in the openshift org has 178 results:
https://github.com/orgs/openshift/repositories?language=&q=o...
Not all are repos hosting one or more microservices, but most appear to be. Best of luck ensuring consistency and quality across so many repos.
Enforcing standards means actually having org level mandates around acceptable development standards, and it's enforced using tools. Those tools should be just as easily run on one monorepo than 50+ distributed repositories, nay?
If you wanted to, sure, applying enough time and money could make it work. I like to think that those resources might be better spent, though.
Why? Is your framework heavily tied to client bindings? APIs I consume occasionally get new fields added to it for data I don't need. My code just ignores it. We also have a policy you cannot add a new mandatory field to API without version bump. So maybe REST API would have new field but I didn't send it and it happily didn't care.
What instead, then? Explicit GitOps. Explicit, reified release specifications (think k8s resource manifests, or Erlang .relup files), one per separately-deploy-cadenced component. If you have a monorepo, then these live also as a dir in the monorepo. CD happens only when these files change.
With this approach, a single PR can atomically merge code and update one or more release specifications (triggering CD for those components), if and when that is a sensible thing to do. But there can also be separate PRs for updating the code vs. "integrating and deploying changes" to a component, if-and-when that is sensible.
How do you avoid the chicken-and-egg problem? Like if the k8s manifest contains a container tag, and the tag is created by CI when the PR is merged to main, it would seem you can’t add code and deploy that code in the same PR.
- PR creation (or any update to the PR branch by a human) would trigger a CI workflow to build+push the container;
- if this succeeds, the workflow would push a commit to the PR feature branch that pins the container ref;
- and the base branch would have a branch protection rule that makes the success of this CI workflow a prerequisite for PR mergeability.
In the explicit-GitOps-in-monorepo case, you'd probably want to do this by having the PR author modify the release spec file of each component they want to build a release for, replacing the old static ref with a temporary symbolic ref meaning "hey CI system, calculate this." Then the CI's added commit would rewrite those temporary symbolic refs into new static ones.
---
Although, that's all assuming that you even want to do CI builds in the first place. If you're developing containerized server software — rather than an operating system, a web browser, etc — then it shouldn't matter where you're building, and there isn't much impetus for deterministic, auditable builds, either. So why bottleneck builds by shoving them all onto some wimpy CI system—when every engineer already has a powerful workstation that can do builds often 4x faster, sitting there idle?
Here's what I call "local-first explicit GitOps":
1. you give your engineers the ability to push refs to the container registry (but you never use symbolic refs, always static manifest SHAs, so a malicious engineer can't do anything dangerous with this.
2. you add a script to your monorepo, that an engineer can run against their feature branch, that'll notice their current branch's local symbolic refs, and build+push containers locally in order to rewrite those into static refs.
3. On success, the script can use `gh` (or equivalent) to trigger PR creation.
Now every PR will come into existence already pinning valid new build artifacts by their static refs!
For example: build code and image, update git reference and version number in Application when build done.
You then get atomic updates of both code and configuration.
The same semantic version string is in pyproject.toml and values.yaml. These are enforced to be the same via pre-commit hook. Environment-specific versions are in env-specific values files like values-prod.yaml.
I have a build workflow which triggers on PRs that change certain files, computes either a pre- or post-release tag by appending the git hash to the server tag, and pushes the images and model repo using that tag. This will re-trigger as the PR is updated.
Users deploy these pre/post-releases in a few clicks with a manual github action which updates the branch and tag in our dev cluster Argo.
The PR can’t be merged unless the version is bumped.
When the PR is merged, a “promote” workflow runs which re-tags the latest image and model repo with the release version (pre-release without git hash suffix). This is fast, which means there’s no risk of image backoff loop from Argo trying to sync before artifacts are available.
Most PRs don’t touch the version in values-prod.yaml, so merges to main auto-deploy to staging while prod requires another PR updating only values-prod.yaml.
Folks complain a bit about manually bumping the version, and needing 2 PRs to take a change to prod, but it’s pretty smooth otherwise!
Of course k8s not required, you can do it in straight IaC etc (i.e deploy a whole parallel system and switch).
It's still "mixed fleet" in terms of any shared external resources (queues, db state, etc) but you can change service interfaces etc with impunity and not worry about compatibility / versioning between services.
Throwing temporary compute at the problem can save a lot of busywork and/or thinking about integration problems.
This stops being practical if you get _very_ big but at that point you presumably have more money and engineers to throw at the problem.
That's overall a bad idea, and negates the whole point of blue-green deployments. This is particularly bad if you have in place any form of distributed transaction.
There are very good reasons why deployment strategies such as rolling deployments and one-box deployments were developed. You need to be able to gradually roll out a change to prevent problems from escalating and causing downtime. If you use all that infrastructure to then flip global switches then you're building up all this infrastructure to the negate any of that with practices that invariably cause problems.
And this is being done just because someone thinks it's a good idea to keep all code in a single repo?
From my perspective I prefer it to rolling updates, it just costs a lot of resources. It gives you a chance to verify all the parts of your system together during release.
You deploy an entire new parallel copy of your system and test it. This can be manual or automated. Once you have confidence, you can flip traffic. Alternatively you can siphon off a slice of production traffic and monitor metrics as you gradually shift traffic over to the new release. Note that you can also nearly "instantly" roll back at any point, unlike with a rolling update. Again, this is only "sort of" a mixed fleet situation because you can maintain the invariant that service A at v1.23 only talks to service B at v1.23. This means you can atomically refactor the interface between two services and deploy them without thinking too hard about what happens when versions are mixed.
Distributed transactions I'm not so sure about, we would have to talk specific situations.
> And this is being done just because someone thinks it's a good idea to keep all code in a single repo?
It's more like, if you have a monorepo this is one option that becomes simpler for you to do, should you happen to want to do it.
Even with a few services, we saw some pretty crunchy issues with people not understanding that service A had version 1.3.1234 of a module and Service B had version 1.3.1245 and that was enough skew to cause problems.
Distinct repos tend to have distinct builds, and sooner or later one of the ten you're building will glitch out and have to be run twice, or the trigger will fail and it won't build at all until the subsequent merge, and having numbers that are close results in a false sense of confidence.
This brings us to the elephant in the room: monorepo strategies are just naive and futile attempts at fixing the problems created by an absolute lack of support for versioning.
If you support versioning, you don't care where and how you store your code. You just push the changes you need to push, and make sure you bump versions to reflect the nature of your changes. Consumers are left with the responsibility of picking which to consume, and when to consume it.
You can't always allow support for V1 of some API to continue because it's just "wrong" in the light of your work. It can turn out that your "micro" architecture just didn't really work because factors X or Y tie certain groups of services together so much that a change to one forces changes to the others.
So redesign right? Well, the contractors are all gone and your small team can barely manage the bugs and minor features, let alone redesigns which tend to force huge changes in the entire system because it wasn't thought out that well to start with. i.e. the mountain to climb to fix a little problem is as high as fixing all the problems.
So you might have to update everything that references your changing API to use the new version....but your company has tons of repos and you don't know which ones reference your API, and they're owned by other teams and there's no central set of tests which have been run against the combination of the head of all these repos to ensure that they do in fact work together at exactly the state the combination is at when you are working.
All deployments are a guess - which repos do you need to deploy for change X? Some other team deploys their repo without realising that it now depends on a new version of yours. Oh dear. We should have created a versioning mechanism of some kind and made their deployment depend on ours......but no time to do that kind of thing and could you get other people to agree to use it and how does it work with your CI system and.........well....you just muddle on, force yourself to remember the details, make mistakes etc etc. Busy work.
IMO monorepos make a good place to put a set of whole-system end to end tests that everyone can run before they merge. That's going to sort out many conflicts before they turn into a bug hunt.
That's how versioning works.
- You allow legacy applications to consume the legacy interface,
- you provide a new interface to the world so that they have a path forward to migrate out of consuming the legacy service through the legacy interface,
- the consumers migrate in a time frame that makes sense to them.
- in the end you sunset the legacy API, and pull the plug.
This scenario is not eliminated by monorepos.
What monorepos do is force consumers to migrate to the new interface as an urgent task that's part of the critical path. If you have resources to do that migration, it does not matter how you store your code.
IMO versioning is something you do when you cannot do the work of upgrading yourself - it's a response you take which is non-optimal and you do it when it's unavoidable.
It doesn't. It just does not force everyone to rush both clients and services to prod without having a fallback plan. Also dubbed as competent engineering.
> We have to be able to make changes on the clients of our code because the teams responsible for them are busy achieving their goals.
Yes. That's what you want, isn't it? Otherwise as a service maintainer you'd be blocked for no reason at all.
And let's not even touch the topic of rolling back changes.
There is no way around it. Once you start to think about any of the rationale behind this monorepo nonsense, you soon realize it's a huge mess: lots of easily avoidable problems created by yourself for yourself, that otherwise would be easy to avoid.
To put the opposite view: Versioning could be done with individual functions in the code for example but we don't do that - we just update all the places that call it.
We usually start to do versioning where there are boundaries - such as with components that we bring in from external sources or with projects that are maintained by separate teams.
A version is usually a response a situation where you cannot update the clients that use the api yourself.
So monorepos can be seen as a way of just saying "actually I can update a lot of the points-of-use of this API and I should instead of creating a new version and waiting for someone else to finally use it."
I think you're a bit confused. There are code repositories, and there are units of deployment. Those are not the same things.
Storing multiple projects in a single repository does not magically make them a single unit of deployment, even when you deploy them all with a single pipeline. When you have multiple projects, you always have multiple units of deployment.
In your example, you focused on a scenario that does not apply: individual functions that are a part of the same unit of deployment. Your example breaks down when you pack your hypothetical function in separate modules that you deploy and consume independently. Even if you do not explicitly tag a version ID to a package, implicitly your module has different releases with different versions of your code delivered at different points in time. If one of these deliveries has a breaking change them your code breaks. Explicitly specifying a version ID, such as adding git modules pointing at a specific commit, is a technique to preserve compatibility.
Where it is very obvious your example fails is when you look at scenarios involving distributed applications with independent units of deployment. This means things like a SPA consuming a backend service, or even a set of producers and consumers. Even if they are all deployed by the same pipeline, you either have forced downtime or you will always have instances of different versions running in parallel.
The SPA+backend is a rather obvious example: even if you do a deployment of a backend and frontend at the exact same time as an atomic transaction that ensures both are available at the precise same tick, don't you still have users with the browsers loaded with instances of the old SPA? They will continue to have it open until they hit F5, aren't they? If you released a breaking change to the backend, what do you think will happen to users still using the old SPA? Things will break, won't they?
Atomic deployments do not exist, however. Looking at services, you have no way to ensure you can deploy new versions of services at precisely the same tick. This means even with monorepos you will always have different deployments of those services running in parallel. Monorepo proponents fool themselves into believing this is not a problem because they count on experiencing problems at best only briefly during deployments, and that the system eventually reaches steady state. This means things like erratic responses, distributed transactions failing, perhaps only a few corrupt records going into the db, etc. If everyone pretends these problems are normal then there is no problem to fix.
Except this negates each and every single hypothetical advantages of a monorepo, and rejects any argument supporting it. Once you realize you're not actually eliminating problems and you are only buying forced downtime, no matter how small, and taking that hit out of willful ignorance. As a tradeoff, you're buying yourself operational problems and lack of flexibility due to the misuse of revision control systems.
And all for what? Because someone heard Google uses monorepos?
It makes it easier to turn them into the same unit of deployment. There's nothing you cannot do some other way of course.
You're right about atomic deployments being difficult and sometimes one can control that risk by the order in which you change things. In a monorepo it's slightly easier to record some kind of script, makefile, dependency system that says "deploy this before that".
With browsers -for sure your user level API has to be stable and cannot change in a sudden incompatible way. When people have seen fit to have layers of APIs underneath it though, one can still have a lot of change thats theoretically hidden from users but still changes lots of APIs.
They are not the same unit of deployment. That's an impossibility.
This critical mistake is at the core of this monorepo nonsense. It's a cargo cult, where people believe that storing code for multiple projects in the same source code revision contol system somehow magically turns distributed systems into a monolith and solves deployment issues. It does not.
> You're right about atomic deployments being difficult and sometimes one can control that risk by the order in which you change things.
No. That is false. Atomicity in a distributed transaction is not achieved by shuffling operations around. Specially those you cannot control.
Not really.
From the producer side it only requires that a) you do not break your contract, b) when you need to break a contract, you bump your version number, c) when no one uses older versions, remove the code.
From the consumer, it's even easier: you just need to point to the version you're consuming.
What exactly do you think is the problem?
Knowing when no one uses older versions is also hard (especially when you can no longer grep for them, unlike a monorepo) so some people get into defensive mode and never delete or fix any interfaces because "someone might still be using it".
I don't see what point you tried to make. So A breaks when someone breaks B. That's ok. You spot the error the moment you try to update A, well before you commit any code. You file a low-priority ticket and go on with your life because A is still good in production.
> Knowing when no one uses older versions is also hard (...)
Not really, it's the easiest part of the whole thing and a non-issue. If you manage a package this does not even register as a problem. If it's a service you have request metrics.
What is the problem, really?
What inevitably happens is that A hasn't been updating their version of B in weeks/months, but some recently landed bug fix in B is now needed RightAway(tm). And now if something accidental breakage happened between now and weeks/months ago, it'll be much more annoying to triage.
> Not really, it's the easiest part of the whole thing and a non-issue. If you manage a package this does not even register as a problem.
Well, where I'm at we have semi-regularly teams we have never heard of using our libraries. How can I know what things such people are using?
A monorepo gives you less control. If you're unable to enforce control through other means, sure use a monorepo. But it's limiting.
No, you just pin your app to v3.2.4 until a working update is made (either to the library or to your own app). I'm not sure why that suddenly becomes an impossible task once the creators of the app and the library work for the same parent company.
> Knowing when no one uses older versions is also hard
That's the thing though - you don't have to care. If you delete an interface, just bump the version number accordingly. People who still need the interface will stay on the old version. (And - bonus - they are now placed under implicit pressure to refactor their code away from using that old interface, now that they know it's deprecated and will no longer be maintained. By contrast, if you have a practice of keeping code around code just because people are still using it, that code tends to just stick around forever.)
External libraries have to be versioned because their code is not part of your monorepo. If the whole world would be pushing in the same monorepo, while not breaking any functionality, we would not need version numbers.
So, like never?
That pretty much makes all internal APIs and code like Public APIs as a product, which adds a huge costs in terms of initial API designs, assumption validations, because you can't make any mistakes at all. And later down the road it fixes you into a place where if new requirements arise, and they will, you won't be able to pivot or refactor anything.
Literally having this version setup and requirements changing I've seen estimates of what would be a week in a monorepo refactor either being estimated to take a year+ or simply impossible because of so many assumptions of those contracts. And there's such a nasty spaghetti of dependencies with different versions that no one can actually understand or hold in their head, and basically to build certain new features, they would have to start from scratch.
No. Different versions are not different products. They are the exact same product, provided in a way that delivers features in a way that won't break clients.
> (...) adds a huge costs in terms of initial API designs (...)
Obviously not. From the producer side, all it takes to support versioning is picking up your versioning strategy and sticking with it. This can mean providing an endpoint through a different path parameter, reading a request header, or even get the version through a query parameter.
Does it take you much design work to add a "v1" in a URL path? How about reading a request header? Does passing a query parameter require meetings to review designs?
Some frameworks even explicitly support versioning as a high-level concept and make it a part of their routing strategies. Copy the source file of a controller to another folder, change it's namespace to avoid conflicts, add a tag to the class, done. You just released a new version of your API.
> (...) assumption validations, because you can't make any mistakes at all.
You somehow managed to get it entirely backwards. With versioning, you can afford making all kinds of mistakes all the time. Breaking interfaces is no longer w concern. You release your changes to an upstream dependency/producer with zero use, downstream dependencies/consumers can gradually switch as their test plans allow, and if any problem is detected then this can be worked out before either consumers go live or producers receive production traffic.
More importantly, you have your old stable versions ready to roll back.
That's the real point of a monorepo: you can treat the whole thing as internal, break it to your heart's content. You only need to make sure you don't break the user-facing part, for some subjective value of break.
There is no dependency hell. Your consumers allocate work effort to migrate to the latest version. This is work you would still have to do with a monorepo.
In my experience, with separate repositories, packages and teams, I think it's a huge challenge to get e.g. 8 different teams together and then to agree to migrate, since everyone's roadmaps are probably full for the following 4 quarters already.
That is, if you’re not tied to using just Github of course.
Big monorepos and multiple repo solutions require some tooling to deal with scaling issues.
What surprises me is the attitude that monorepos are the right solution to these challenges. For some projects it makes sense yes, but it’s clear to me that we should have a solution that allows repositories to be composed/combined in elegant ways. Multi-repository pull requests should be a first class feature of any serious source code management system. If you start two projects separately and then later find out you need to combine their history and work with them as if they were one repository, you shouldn’t be forced to restructure the repositories.
> Multi-repository pull requests should be a first class feature of any serious source code management system.
> If you start two projects separately and then later find out you need to combine their history and work with them as if they were one repository, you shouldn’t be forced to restructure the repositories.
Not in my experience. I use "git blame" all the time, and routinely read through commits from many years ago in order to understand why a particular method works the way it does.
Luckily, there are many tools for merging git repos into each other while preserving history. It's not as simple as copy and paste, but it's worth the extra efford.
Do you have examples of source code management systems that provide this feature and do you have experience with them? repo-centric approach of GitHub often feels limiting.
Having a centralized configuration library (a shared Makefile that we can pull down into our repo and include into the local Makefile) helps, until you have to make a backwards-incompatible change to that Makefile and then post PRs to every branch of every repo that uses that Makefile.
Now we have almost the entirety of our projects back into one repository and everything is simpler; one PR per release branch, three PRs (typically) for any change that needs backporting. Vastly simpler process and much less room for error.
As soon as you split 1 repo into 2 repos you need to start building tooling to support your 2 repos. If your infrastructure is sufficiently robust with 2 repos then you might as well have 3 or 4 or 10. If it's built to _only_ support 2 repos (or 3 or 4) then it's brittle out of the gate.
The value of a monorepo is that you completely eliminate certain classes of problems and take on other classes of problems. Classic trade off. Folks that prefer monorepos take the position that multirepo problems are much harder than monorepo problems most of the time.
You can have as many build targets in a monorepo as you like. You can also have large monoliths in a monorepo.
No, not really.
If you're talking about projects for modules and components, all you need is a versioning strategy and release consumable packages of your projects.
If you're talking about services, all you need to do is support versioned APIs and preserve your contracts.
No tooling required. For projects you can even make do with git submodules. For services, all you need to do is update clients of your downstream dependencies.
What problems are you facing, exactly?
If you aren't using a monorepo, you need some versioning process, as well as procedural systems in place to ensure that everyone's dependencies, stay reasonably up to date. Otherwise, you end up deferring pain in really unwanted ways, and require sudden, unwanted upgrades through api incompatibility due to external pressure.
This also has the downside of allowing api-owning teams to make changes willy-nilly and break backwards compatibility because they can just do it behind SemVer, and then clients of the api need to own the process of migrating to the new version.
A monorepo fixes both of these: you cannot get out of sync, so it is the api-owning team's responsibility to upgrade clients, since they can't break the API otherwise. Similarly, you get a versioning process for free, and clients can never be using out of date or out of support versions of a dependency.
Services work approximately the same either way, since you can't assume synchronous upgrades across service/rpc boundaries anyway.
The fundamental mismatch is the feature/app code will have longer testing times, stuff like Puppeteer and creating-infra-per-MR that just fundamentally takes a long time to run. But in ops, you need configuration to roll out quickly, maybe run an autoformatter and a linter or two beforehand and that's it. When you want to rollback, you don't need to wait for all your tests to run.
Yes you need to deal with versioning. You can just use automatic timestamps. You can write easy automation to push new timestamps to the ops/config repository when there are new releases in the app repository. The ops repository has minimal configuration to pull the latest version of the app repository and apply it, where the app repository includes infra and deployment scripts themselves.
What are the solutions? You would need multiple merge queues.
If you try to come up with fancy rules like forbidding changes to infra, config, and code in the same commit, then you're just cramming a polyrepo design into a monorepo, with much more fragility, because commits get rebased and squashed and etc.
Wrote this some years ago: https://dev.to/kgunnerud/our-experience-monorepo-with-java-m...
Nobody in our team wants to go back to nonmonorepo now, although everone was sceptical initially
Primary monorepo – single versioned packages for libraries and services that are deployed as a whole.
Secondary monorepo – individually versioned packages for shared libraries and independent (micro)services.
You only need to coordinate this sort of changes if you have absolutely no versioning strategy in place and accept breaking contracts for no reason at all.
If you have any form of versioning in place, you do not need to push multi-project commits. You update upstream packages and services, roll them out, and proceed to update the projects that consume them to gradually point them to the newly released versions.
If you're consuming a module from a git as a submodule, yes.
Same idea if you use some kind of versioning and release system. It's still a bit of a pain with all the PRs and coordination involved, but at every step every branch is consistent and buildable, you just check it out and hit build.
This is obviously more difficult if you have a more loosely coupled architecture like microservices. But that's self-inflicted pain
Your software artifacts will have loose coupling if you divided them well enough on their creation. As soon as they are created, you can't do anything else to change it, except for joining or splitting them.
Now, though, it seems the pendulum has swung in the opposite direction, from “everything in its own repo” to “everything in one repo.” This, too, will create its own set of problems, which also can be comedic, but frustrating to experience. For instance, what happens when someone accidentally pushes a certificate or API key and you need to force an update upstream? Coordinating that with 50 developers spread across 8 projects, all in a single repo.
Instead we could also face the problems we currently face and start out wirn a balanced approach. Start with one repository, or split frontend and backend if needed. For data pipelines that share models with the API, keep them in the same repository, creating a single source of truth for the data model. This method has often led to other developers telling me about the supposed benefits of “everything in its own repo.” Just as I pushed back then, I feel the need to push back now against the monorepo trend.
The same can be said for monoliths and microservices, where the middle ground is often overlooked in discussions about best practices.
They all reminded me of the concept of “no silver bullet”[0]. Any decision will face its own unique challenges. But silver bullet solution can create artificial challenges that are wasteful, painful, and most of all unnecessary.
The correct approach here is typically to invalidate the certificate or API key. A force push usually doesn't work.
If you're using GitHub, the dangerous commit lives on effectively forever in an awkward "not in a repository" state. Even if you're not on GitHub and your system actually garbage collects, the repo has been cloned onto enough build machines and dev machines that you're better off just treating the key or cert as compromised than trying to track down all the places where it might have been copied.
You’re correct about keys/certs once uploaded, they should be treated as compromised, especially when the repository isn’t self-hosted. However, replacing API keys and certificates can take time, and within a large corporation, it could take months.
Force pushes occur for various reasons. Sensitive data includes customer and employee personal info.
Other cases involve logs, DB backups, cache, PDFs, configs, and binary files.
Maintenance and performance needs form another category.
Team dynamics and varying git expertise can lead to accidental or uninformed force pushes, especially challenging in a monorepo with 50+ contributors.
In summary, reasons range from personal data (GDPR compliance), security requirements for logs/configs, to resolving merge conflicts and performance issues.
Your question regarding the need to communicate between 50 or more devs, there was no need but monorepo idea forces unnecessary communication and team effort when none was needed if there would be more repositories.
This scenario only happens if you're working with a distributed monolith, you care nothing about breaking existing APIs, and you own all producers and consumers.
When this happens, your problem is obviously not how you maintain projects independently in separate repos. Your problems is how you're failing to preserve API contracts, and instead you go to the extreme of insisting in breaking all contracts at once.
It's also a huge red flag when you talk about "accidentally end up with one repo in this branch". Don't you know what you are pushing to production?
As always, monorepo talks are just a thin veil of optimization placed over a huge mess of major operational and technical problems.
Preserving contracts has a significant cost. Why would you try to preserve contracts if you're in fact in control of all producers and consumers?
Not really. You just need to break out a new version when you put out a breaking change. This can mean anything between duplicating the code that implements your interface to then apply a change and checking a version flag when invoking an operation to determine which version to run.
Do you struggle to not break your code? To me it's like the main responsibility of any developer, isn't it?
> Why would you try to preserve contracts if you're in fact in control of all producers and consumers?
Because you clearly aren't in control, if you feel compelled to dump anything and everything in the same repository to try to prevent things from breaking.
Also, as others have pointed out, storing code in the same repository gives you no assurance that all your services will be deployed at the same exact moment in time. Consider for example multi-region deployments, or even deployments to multiple availability zones. Those aren't atomic. Why does anyone believe that storing all code in the same repository is enough to make these operations atomic?
Not isolated to this particular issue. People really like to pretend that problems are much simpler than they actually are and that inherent complexity is just some edge cases to be ironed out.
On one end you have Netflix-like architectures, on the other end you have intranet apps used twice monthly by two users. There's a wide range of needs and expectations in between.
Yes, but my time is limited and there's the opportunity cost.
It also means more code, more tests for future. More code, ceteris paribus, represents more maintenance costs for the future.
> Because you clearly aren't in control, if you feel compelled to dump anything and everything in the same repository to try to prevent things from breaking.
I can prevent things from breaking in multi-repo, multi-versioned architecture, it's just more expensive to do so.
No one is saying that. But there are lots of types of shared code that can be updated atomically, so why not update them atomically? And there's nothing inherent in being a monorepo that prevents you from being careful about the non-atomic updates.
It's really easy to accidentally deploy multiple repos in a way that gets the deployment order wrong. You have to be careful with non-atomic deployments whatever type of repo structure you have.
Of course I’m “breaking all contracts at once”, for most of them were never a contract to begin with, and the monorepo effectively discourages people from assuming random behaviors are in fact contracts, and that preserves my sanity and the sanity of the code base that now doesn’t suffer from the combinatorial explosion in complexity that happens when people want everything to be an API contract.
Yes, they are. They are as free as adding a new endpoint.
> and for things that were not decided to be an API and or a contract an absolute waste of time.
You're talking about integrating multiple projects. There are always contract. You simply chose to be ignorant of that fact and fail to interpret breaking those contracts as the root cause of all your problems.
You point out anyone lauding monorepos as some kind of solution to any problem and I assure you that we can immediately isolate one or more problems caused by versioning and breaking contracts.
The only reason why we see this nonsense paraded in web services is that they are loosely coupled. When people worked mainly with modules and libraries, compiler and linking errors made this sort of error very obvious. Now people chose to unlearn facts and want to call that operational improvements.
This reminds me a lot of the dynamic vs. static typing discussion. Even at a function level, there's always a contract, no other option. The only decision you can make is whether it should be well documented, compiler verifiable and explicit, or not.
Tooling issue. Bitbucket will happily link PRs if instructed so.
> prevents a lot of mistakes from not having the right revision synced down across different repos.
Tooling issue. Depending on setup, "tooling" could be few lines in your favorite scripting language.
---
Monorepo steps around some tooling issues, but sacrifices access control, checkout speed
Some benefits include:
* Separate repos evolve better than a monorepo. Particularly in early stage companies where pivots are frequent. * They keep tools just doing one thing well and prevent tying everything in the codebase together. * Dependencies are more flexible, this is particularly useful with ML libraries. * A lot of repo tooling is simpler with medium sized single purpose projects * Stronger sense of ownership boundaries
When I've encountered this, we've had to use another repo to keep scripts that managed this. But this was also sometimes problematic because each developer's setup had to be identical on their local file system (for the script to work) or we had to each create a config file pointing to where each repo lived.
This also impacts tracking down bugs and regression analysis; this is much easier to manage in a mono-repo setup because you can get everything at the same revision instead of managing synchronization of multiple repos to figure out where something broke.