News for hackers. Stuff that matters.
Re:Monorepoize – Bash scripts for creating a monorepo out of... by bluGill on Monday April 27, 2020 @08:18AM (#22995094)
If that happens often you have the wrong repo split. This isn't a condemnation of the multi repo approach, only your architecture.

Note that this is about trade offs. If you have a monorepo this problem goes away but now you need to manage the problems of monorepos. If you have multiple repos you get this and other problems instead. Pick the right tradeoffs for your own needs. The only things wrong is claiming your answer is the right one for everyone else.

Re:Monorepoize – Bash scripts for creating a monorepo out of... by auxym on Monday April 27, 2020 @09:09AM (#22995448)
> If that happens often you have the wrong repo split.

Yes, but: you pick out a nice, clean perfectly segregated architecture. Two years later, things have evolved, and a new feature that no one initially thought of is asked and involved refactoring 25 repos. Oops.

Side note: not my architecture. I only worked there for a short while. I have no skin in the game either way, I'm sure I agree with you that mono and multi- repos are the right solution for different problems.

Just trying to expose problems I've seen happen in a massive multirepo approach. Everyone thought it was the correct solution after being burned by a massive monolith that over two decades became an entangled mess. They thought they had the perfect architecture, the "correct repo split". In practice it's almost impossible, which is a major tradeoff to be considered.

Minimizin the number of repos and splitting only when absolutely necessary at least has a chance of reducing that complexity. The default should be "as few repos as possible", not "we're going multirepo, let us break this down into as many small parts as possible as our first step", which might be what the architects were used to in modular program architecture (small, reusable functions/classes).

Re:Monorepoize – Bash scripts for creating a monorepo out of... by vorpalhex on Monday April 27, 2020 @10:25AM (#22996094)
> The default should be "as few repos as possible"

In any corporate/enterprise-y environment with multiple teams, the default should be the smallest piece likely to change ownership.

When we see a pattern where a feature involves refactoring a bunch of repos, then your initial system breakdown was horrendously off-track and you lost sight of your domains. Been there, it is fixable but it takes a bit.

> Everyone thought it was the correct solution

There is no correct solution, there is the best solution for your problem set. Sometimes that's a monolith, and sometimes that's several small repos. Every once in a while, it's even a monorepo.

Re:Monorepoize – Bash scripts for creating a monorepo out of... by throwaway894345 on Monday April 27, 2020 @11:14AM (#22996602)
Even with the best architecture, you probably have some shared libraries that are used across teams. These will often need to be updated, and they will affect several other packages (other repos, in the multirepo case). The only way to avoid this is to duplicate all code. Moreover, you can have a great architecture, but your teams are "feature-oriented" instead of service oriented (or some other vertically oriented team structure) in which case they'll need to touch many different repos (I don't imagine feature-oriented repos are a good idea) per ticket.

I'm increasingly convinced that the monorepo approach is the right one, but the tooling needs to get better. Bazel is a hydra of incidental complexity and Nix has the right core idea but suffers from a variety of other problems (poor documentation, gratuitously and unnecessarily unfamiliar, too few escape hatches, etc). These issues aren't fundamental to the domain, and there's no reason these tools couldn't change course nor is there a reason that another tool couldn't come in and eat their lunch.

Re:Monorepoize – Bash scripts for creating a monorepo out of... by beagle3 on Monday April 27, 2020 @10:22AM (#22996066)
> you need to manage the problems of monorepos

If have not seen a problem on monorepos that does not exist on multirepos, other than more quickly reaching tools' repo size limits (but this is becoming less of a problem all the time on e.g. git and mercurial, and supposedly was never a problem with perforce).

I have seen many problems multirepos have that monorepos do not - such as that multirepos can (and almost always do) have a wrong split, and monorepos by definition cannot.

Re:Monorepoize – Bash scripts for creating a monorepo out of... by yokaze on Monday April 27, 2020 @03:54PM (#22999317)
> If have not seen a problem on monorepos that does not exist on multirepo

How do you refer to a library in a monorepo from a different repo?

The workflow seems to me to entail an all-or-nothing approach.

Re:Monorepoize – Bash scripts for creating a monorepo out of... by beagle3 on Monday April 27, 2020 @04:21PM (#22999582)
I never tried it, but if I needed it, I would easier use sparse submodule checkout (if it’s possible - I don’t know if that’s true), or submodule the entire repo, and the symbolic link into the internal part I need.

If all else fails (e.g. you need 50 files out of a 100GB repo, which makes things too slow), I would set up a git subtree extract as a checkin hook, which would mean there exists another repo/branch automatically updated on checkin - and which you could use any way you want, e.g, simple submodule.

Do note that’s just what I said: the problem, “I only need a small part of the repo” is independent of it being a monorepo or multirepo.

You would only avoid this problem in a multirepo if you cut it to pieces in exactly the right places. it’s just a matter of whether the parts of the repo you don’t need are too large to make this comfortable — more likely in a monorepo as it is larger - but will happen in a big enough multirepo as well.