News for hackers. Stuff that matters.

How do people manage hundreds of Git repositories?

I have worked with several large companies using git as their version control system. One hosts several hundred git repositories in bitbucket stash, the other has around a hundred repositories in github. We recently ran into an issue with joda-time where upgrading past java8u60 caused apps to fail on startup. So my team is tasked with getting all of our microservices working with the latest jdk.

What tools are there to modify and commit similar changes across 100+ repositories at once?

I have found gitwalk https://www.npmjs.com/package/gitwalk which does a subset of what I want focused on github. But google has not revealed many other tools that can do discovery and transformations across 100s of repositories.

( 15 comments | 17 points | on HN )
Re:How do people manage hundreds of Git repositories? by ericfrederich on Wednesday October 04, 2017 @10:06AM (#15400530)
This looks like a one-off thing you need to do semi-manually and methodically.

Every step of the way you need to balance the effort of doing something with a script vs doing it manually.

How long would it take to manually make the change in all 100+ repos... vs. how long would it take to write a script to do it. Would the script be robust enough to properly make the change in like 90% of them... etc.

These are things you need to ask yourself in your endeavor.

You'd want to do all this work on a new similarly named branch (or branches depending on your workflow) for every repo.

Perhaps write a script to validate changes to all 100+ repos on that branch. Then when all the work is done... do a massive merge.

Re:How do people manage hundreds of Git repositories? by tedmiston on Tuesday October 03, 2017 @11:22PM (#15398073)
It probably depends on exactly what and how complex your “transformation” is. You definitely can do lots of automation with $ git itself in terms of committing, merging, opening PRs on GitHub via $ hub, etc.

Another idea is if you can refactor the common bits into a shared standard library, for instance the design pattern is something like a system with a bunch of plugins, then most of the plugin repos become very tiny. Ideally you’d also have some sort of infrastructure automation around deployment.

Re:How do people manage hundreds of Git repositories? by Sevii on Wednesday October 04, 2017 @12:12AM (#15398202)
We have shared libraries and maven plugins setup. This is a situation where I really wish we used a monorepo and I could just do one commit across 100 microservices using grep and other tools to update pom.xml files. I have read whitepapers from google on how they support this kind of operation, but I need it for many git repos.
Re:How do people manage hundreds of Git repositories? by ynezz on Wednesday October 04, 2017 @02:30AM (#15398648)
This sounds like StackOverflow question. Anyway, if you're proficient in Java, why don't you write small app which will iterate over all Git repo dirs, modifies the files, makes commit, push and creates PRs? Not a big deal, right?
Re:How do people manage hundreds of Git repositories? by Sevii on Wednesday October 04, 2017 @10:38AM (#15400732)
That is the current plan, combined with plenty of manual work verifying jenkins jobs and deployments.
Re:How do people manage hundreds of Git repositories? by assafmo on Wednesday October 04, 2017 @01:45PM (#15402428)
If you can do it if there was just one repo, then juat clone those 100 repos under the same parent directory and execute the script in the parent directory
Re:How do people manage hundreds of Git repositories? by assafmo on Wednesday October 04, 2017 @03:39PM (#15403804)
and then in a loop commit the changes
Re:How do people manage hundreds of Git repositories? by sethish on Wednesday October 04, 2017 @09:16AM (#15400212)
I haven't found anything off-the-shelf that helps. I manage a project with > 50k repositories in a GitHub organization. For the past few years I've been developing software that helps with my [particular goals](https://github.com/gitenberg-dev/gitberg), but I'm afraid there isn't much there that may help you.
Re:How do people manage hundreds of Git repositories? by tenken on Wednesday October 04, 2017 @12:23AM (#15398237)
Have you looked at a tool like: https://github.com/mixu/gr
Re:How do people manage hundreds of Git repositories? by Sevii on Wednesday October 04, 2017 @12:55AM (#15398345)
Going to look at it, thanks.
Re:How do people manage hundreds of Git repositories? by zeveb on Wednesday October 04, 2017 @05:58PM (#15405078)
It's too late for you, but I think that it's useful for other folks to see your example and realise why multiple repos are a bad idea.

In the general case, each organisation should have a single repo containing all of its code. There are times to violate this rule, but unless you're in extraordinary circumstances, more than a dozen repos in an organisation is an extremely bad mistake.

Re:How do people manage hundreds of Git repositories? by allwein on Thursday October 05, 2017 @11:17AM (#15409853)
Can you elaborate on this a bit? What are the advantages to keeping everything in a single repo for the organization vs. a different repo per project?
Re:How do people manage hundreds of Git repositories? by zeveb on Thursday October 05, 2017 @03:12PM (#15411797)
Basically, having multiple repos means that one person can change a dependency of someone else's code without fixing the dependent code. This is unavoidable with independent organisations, but is easily avoidable within an organisation which uses unit & integration testing within a single repo (the use of a single repo means that a suite of changes may be applied atomically).

In general, a process which allows one person in an organisation to stymie another's progress is a broken system.

Re:How do people manage hundreds of Git repositories? by codeonfire on Wednesday October 04, 2017 @04:16AM (#15398996)
Grep + sed or replace in Files for the changes. And gitall to commit them all. There is no need to get fancy if simple tools will do the job.
Re:How do people manage hundreds of Git repositories? by frou_dh on Wednesday October 04, 2017 @05:16PM (#15404776)
`mr` is good for keeping track of, and working with, many repos in bulk.

http://myrepos.branchable.com/

IIRC it has a good man page explaining more advanced use.