News for hackers. Stuff that matters.
Re:The largest Git repo by js2 on Wednesday May 24, 2017 @02:23PM (#14412242)
Windows, because of the size of the team and the nature of the work, often has VERY large merges across branches (10,000’s of changes with 1,000’s of conflicts).

At a former startup, our product was built on Chromium. As the build/release engineer, one of my daily responsibilities was merging Chromium's changes with ours.

Just performing the merge and conflict resolution was anywhere from 5 minutes to an hour of my time. Ensuring the code compiled was another 5 minutes to an hour. If someone on the Chromium team had significantly refactored a component, which typically occurred every couple weeks, I knew half my day was going to be spent dealing with the refactor.

The Chromium team at the time was many dozens of engineers, landing on the order of a hundred commits per day. Our team was a dozen engineers landing maybe a couple dozen commits daily. A large merge might have on the order of 100 conflicts, but typically it was just a dozen or so conflicts.

Which is to say: I don't understand how it's possible to deal with a merge that has 1k conflicts across 10k changes. How often does this occur? How many people are responsible for handling the merge? Do you have a way to distribute the conflict resolution across multiple engineers, and if so, how? And why don't you aim for more frequent merges so that the conflicts aren't so large?

(And also, your merge tool must be incredible. I assume it displays a three-way diff and provides an easy way to look at the history of both the left and right sides from the merge base up to the merge, along with showing which engineer(s) performed the change(s) on both sides. I found this essential many times for dealing with conflicts, and used a mix of the git CLI and Xcode's opendiff, which was one of the few at the time that would display a proper three-way diff.)

Re:The largest Git repo by Peaker on Wednesday May 24, 2017 @02:59PM (#14412534)
When you have that many conflicts, it's often due to massive renames, or just code moves.

If you use git-mediate[1], you can re-apply those massive changes on the conflicted state, run git-mediate - and the conflicts get resolved.

For example: if you have 300 conflicts due to some massive rename, you can type in:

  git-search-replace.py[2] -f oldGlobalName///newGlobalName
  git-mediate -d
  Succcessfully resolved 377 conflicts and failed resolving 1 conflict.
  <1 remaining conflict shown as 2 diffs here representing the 2 changes>
[1] https://medium.com/@yairchu/how-git-mediate-made-me-stop-fea...

[2] https://github.com/da-x/git-search-replace

Re:The largest Git repo by dom0 on Wednesday May 24, 2017 @04:13PM (#14413074)
Also git rerere

When maintaining multiple release lines and moving fixes between them:

Don't use a bad branching model. Things like "merging upwards" (=committing fixes to the oldest branch requiring the fix, then merging the oldest branch into the next older branch etc.), which seems to be somewhat popular, just don't scale, don't work very well, and produce near-unreadable histories. They also incentivise developing on maintenance branches (ick).

Instead, don't do merges between branches. Everything goes into master/dev, except stuff that really doesn't need to go there (e.g. a fix that only affects a specific branch(es)). Then cherry pick them into the maintenance branches.

Re:The largest Git repo by seanp2k2 on Wednesday May 24, 2017 @08:13PM (#14414443)
Cherry picking hotfixes into maint branches is cool until you have stuff like diverging APIs or refactored modules between branches. I don't know of a better solution; it kind of requires understanding in detail what the fix does and how it does it, then knowing if that's directly applicable to every release which needs to be patched.
Re:The largest Git repo by jjawssd on Wednesday May 24, 2017 @09:51PM (#14414801)
Use namespaces to separate API versions?

POST /v4/whatever

POST /v3/whatever

Re:The largest Git repo by _asummers on Friday May 26, 2017 @04:21PM (#14426847)
We version each of our individual resources. so a /v1/user might have many /v3/post . Seems to work for us as a smaller engineering team.
Re:The largest Git repo by jjawssd on Tuesday May 30, 2017 @12:10PM (#14446303)
A better approach would be to alias /v3/user to /v1/user until there is a breaking change needed in the v3 code tier.
Re:The largest Git repo by _asummers on Thursday June 01, 2017 @08:17AM (#14459621)
On a rapidly developing API, that would be way too much churn on our front end. For an externally facing API, I completely agree.
Re:The largest Git repo by dom0 on Thursday May 25, 2017 @03:35AM (#14415879)
Yes, although this applies to all forms of porting changesets or patches between branches or releases.
Re:The largest Git repo by lmitchell on Wednesday May 24, 2017 @11:30PM (#14415128)
I don't understand the hate for merging up. I've worked with the 'cherry-pick into release branches' model, and also with an automated merge-upwards model, and I found the automerge to be WAY easier to deal with. If you make sure your automerge is integrated into your build system, so a failing automerge is a red build that sends emails to the responsible engineers, I found that doing it this way removed a ton of the work that was necessary for cherry-picking. I can understand not liking the slightly-messier history you get, but IMO it was vastly better. Do you have other problems with it, or just 'unreadable' histories and work happening on release branches? Seems like a good trade to me.
Re:The largest Git repo by dom0 on Thursday May 25, 2017 @03:33AM (#14415871)
As the branches diverge, merges take more and more time to do (up to a couple hours, at which point we abandoned the model)... they won't be done automatically. Since merges are basically context-free it's hard to determine the "logic" of changed lines. Since merges always contain a bunch of changes, all have to be resolved before they can be tested, and tracing failures back to specific conflict resolutions takes extra time. Reviewing a merge is seriously difficult. Mismerges are also far more likely to go unnoticed in a large merge compared to a smaller cherry pick. With cherry picking you are only considering one change, and you know which one. You only have to resolve that one change, and can then test, if you feel that's necessary, or move on to the next change.

Also; https://news.ycombinator.com/item?id=14413681

I also observed that getting rid of merging upwards moved the focus of development back to the actual development version (=master), where it belongs.

Re:The largest Git repo by eridius on Wednesday May 24, 2017 @03:18PM (#14412697)
I just looked at git-mediate and I'm very confused. It appears that all it does is remove the conflict markers from the file after you've already manually fixed the conflict. Except you need to do more work than normal, because you need to apply one of the changes not only to the other branch's version but also to the base. What am I missing here, why would I actually want to use git-mediate when I'm already doing all the work of resolving the conflicts anyway?
Re:The largest Git repo by amag on Wednesday May 24, 2017 @04:44PM (#14413302)
It looks like git-mediate does one more important thing; it checks that the conflict is actually solved. In my experience it's very easy to miss something when manually resolving a conflict and often the choices the merge tools give you are not the ones you want.
Re:The largest Git repo by deepsun on Wednesday May 24, 2017 @05:46PM (#14413724)
IntelliJ IDEA default merger knows when all conflicts in a file are handled and shows you a handy notification on top "Save changes and finish merging".
Re:The largest Git repo by mandeepj on Friday May 26, 2017 @03:50PM (#14426664)
TFS's conflict resolver also does this
Re:The largest Git repo by eridius on Wednesday May 24, 2017 @09:25PM (#14414709)
Well, all it checks is that you modified the base case to look like one of the two other cases. That doesn't actually tell you if you resolved the conflict though, just that you copied one of the cases over the base case.
Re:The largest Git repo by Peaker on Friday May 26, 2017 @03:28AM (#14422508)
True, but if you follow a simple mechanical guideline: Apply the change 2 other versions (preferably the base one last) - then your conflict resolutions are going to be correct.

From experience with many developers using this method, conflict resolution errors went down to virtually zero, and conflict resolution time has improved by 5x-10x.

Re:The largest Git repo by eridius on Friday May 26, 2017 @01:16PM (#14425654)
There's a much simpler mechanical guideline that works without git-mediate: Apply the change to the other version. git-mediate requires you to apply the change twice, but normal conflict resolution only requires you to apply it once.
Re:The largest Git repo by Peaker on Saturday May 27, 2017 @08:35AM (#14429841)
Except you don't really know if you actually applied the full change to the other version. That's what applying it to the base is all about.

You often take the apparent diff, apply it to the other version, and then git-mediate tells you "oops, you forgot to also apply this change". And this is one of the big sources of bugs that stem from conflict resolutions.

Another nice thing about git-mediate is that it lets you safely do the conflict resolution automatically, e.g: via applying a big rename as I showed in the example, and seeing how many conflicts disappear. This is much safer than manually resolving.

Re:The largest Git repo by eridius on Saturday May 27, 2017 @04:39PM (#14431847)
Applying the change to the base doesn't prove that you applied the change to the other version. It only proves that you did the trivial thing of copying one version over the base. That's kinda the whole point of my complaint here, git-mediate is literally just having you do busy-work as a way of saying "I think I've applied this change", and that busy-work has literally no redeeming value because it's simply thrown away by git-mediate. Since git-mediate can't actually tell if you applied the change to the other version correctly, you're getting no real benefit compared to just deleting the conflict markers yourself.

The only scenario in which i can see git-mediate working is if you don't actually resolve conflicts at all but instead just do a project-wide search&replace, but that's only going to handle really trivial conflicts, and even then if you're not actually looking at the conflict you run the risk of having the search & replace not actually do what it's supposed to do (e.g. catching something it shouldn't).

Re:The largest Git repo by Peaker on Sunday May 28, 2017 @03:20AM (#14433441)
> Since git-mediate can't actually tell if you applied the change to the other version correctly, you're getting no real benefit compared to just deleting the conflict markers yourself

This is patently and empirically false:

1) See the automated rename example. How do you gain the same safety and ease of resolution without git mediate? This is, unlike you say, an incredibly common scenario. After all, conflicts are usually due to very wide, mechanical changes. The exact kinds of changes that are easy to re-apply project-wide. It is true for not only renames, but also whitespace fixes which are infamous for causing conflicts and thus inserting bugs, and due to that are not allowed in many collaborative projects!

2) Instead of having to tediously compare the 3 versions to make sure you haven't missed any change when resolving (a very common error!) you now have to follow a simple guideline: Apply the same change to 2 versions.

This guideline is simple enough to virtually never fuck it up, unlike traditional conflict resolution which is incredibly error-prone. Your complaint that git mediate does not validate you followed this one guideline is moot, since this guideline is so easy to not fuck up - compared to the rest of the resolution process.

If you follow this guideline - git-mediate has tremendous value. It makes sure you did not forget any part of the change done by either side - and streamlines every other part of the conflict resolution process:

A) Takes my favorite editor directly to the conflict line

B) Shows me 2 diffs instead of 3 walls of text

C) Lets me choose the simpler diff to apply to the other sides, making a very minimal text change in my editor.

D) Validates that the change I applied was the last one (or decreases the size of the diff otherwise)

E) Does the "git add" for me, and takes me directly to the next conflict

This has been used by dozens of people, who can all testify that it:

A) Made conflict resolution easy and convenient

B) Reduced the error rate to zero (I don't remember the lasts bug inserted in a merge conflict)

C) Sped the process up by an order of magnitude

Re:The largest Git repo by eridius on Sunday May 28, 2017 @03:45PM (#14435786)
> See the automated rename example. How do you gain the same safety and ease of resolution without git mediate? This is, unlike you say, an incredibly common scenario.

Is it? I'm not sure if I've ever had a conflict that would be resolved by a global find & replace. Globally renaming symbols isn't really all that common. In my experience conflicts are not "usually due to very wide, mechanical changes", they're due to two people modifying the same file at the same time.

> It is true for not only renames, but also whitespace fixes which are infamous for causing conflicts …

Most projects don't go doing whitespace fixes over and over again. In projects that do any sort of project-wide whitespace fixes, that sort of thing is usually done once, at which point the whitespace rules are enforced on new commits. So yes, global whitespace changes can cause commits, but they're rather rare.

> Instead of having to tediously compare the 3 versions to make sure you haven't missed any change when resolving (a very common error!) you now have to follow a simple guideline: Apply the same change to 2 versions.

> This guideline is simple enough to virtually never fuck it up

You know what's even simpler? "Apply the same change to 1 version". Saying "Fix the conflict, and then do extra busy-work on top of it" is not even remotely "simpler". It's literally twice the amount of work.

The only thing git-mediate appears to do is tell you if a project-wide find&replace was sufficient to resolve your conflict (and that's assuming the project-wide find&replace is even safe to do, as I mentioned in my previous comment). If you're not doing project-wide find&replaces, then git-mediate just makes your job harder.

From reading your "process" it appears that all you really need is a good merge tool, because most of what you describe as advantageous is what you'd get anyway when using any sort of reasonable tool (e.g. jumping between conflicts, showing correct diffs, making it easy to copy lines from one diff into another, making it easy to mark the hunk as resolved).

Re:The largest Git repo by Peaker on Monday May 29, 2017 @02:36AM (#14437731)
> In my experience conflicts are not "usually due to very wide, mechanical changes", they're due to two people modifying the same file at the same time.

Then it is likely a reversal of cause & effect. People are more reluctant to make wide sweeping changes such as renames because they're worried about the ensuing conflicts.

> Most projects don't go doing whitespace fixes over and over again

Again, for similar reasons. Projects limp around with broken indentation (tabs/spaces), trailing whitespaces, dos newlines, etc - because fixing whitespace is against policy. Why? Conflicts.

> You know what's even simpler? "Apply the same change to 1 version".

It sounds simpler, but not fucking it up is not simple at all, as evidenced by conflicts being a constant source of bugs.

When you look at the 3 versions, and "apply the diff to 1 version" you actually think you apply the diff to 1 version. You're applying the perceived diff to 1 version, which often differs from the actual diff as it may include subtle differences that aren't easily visible. This is detected by git-mediate the double-accounting of applying the perceived diff to both - validating that the perceived diff equals the actual diff.

Without git-mediate? At best you bring up build errors. At worst, revive old bugs that were subtly fixed in the diff you think you applied.

> then git-mediate just makes your job harder.

You're talking out of your ass here. Me and 20 other people have been using git-mediate and it's been a huge game changer for conflict resolution. Every single user I've talked to claims huge productivity/reliability benefits from using it.

> all you really need is a good merge tool

I've used "good" merge tools a lot. They're incredibly inferior to my favorite text editor:

* Text editing within them is tedious and terrible

* Their supported actions of copying whole lines from one version to the other are useless 90% of the time.

Let me ask you this:

What percentage of the big conflicts you resolve with a "good merge tool" - build & run correctly on the first run after resolution?

For git-mediate, that is easily >99%.

Re:The largest Git repo by eridius on Monday May 29, 2017 @07:43PM (#14442120)
> People are more reluctant to make wide sweeping changes such as renames because they're worried about the ensuing conflicts.

I disagree. People generally don't do project-wide find&replaces because it's just not all that common to want to rename a global symbol.

> Projects limp around with broken indentation (tabs/spaces), trailing whitespaces, dos newlines, etc - because fixing whitespace is against policy. Why? Conflicts.

You seem to have completely missed the point of my comment. I'm not saying projects limp along with bad whitespace. I'm saying projects that decide upon whitespace rules typically do a single global fixup and then simply enforce whitespace rules on all new commits. That means you only ever have one conflict set due to applying whitespace policy, rather than doing it over and over again as you suggested.

> You're applying the perceived diff to 1 version, which often differs from the actual diff as it may include subtle differences that aren't easily visible.

Then you're using really shitty merge software. Any halfway-decent merge tool will highlight all the differences for you.

> Without git-mediate? At best you bring up build errors. At worst, revive old bugs that were subtly fixed in the diff you think you applied.

And this is just FUD.

It's simple. Just stop using Notepad.exe to handle your merge conflicts and use an actual merge tool. It's pretty hard to miss a change when the tool highlights all the changes for you.

> You're talking out of your ass here.

And you're being uncivil. I'm done with this conversation.

Re:The largest Git repo by Peaker on Thursday May 25, 2017 @01:11PM (#14418597)
That's why I showed an example of a rename. You write "manually fixed the conflict", where do you see that in the rename example?

You just re-apply either side of the changes in the conflict (Base->A, or Base->B) and the conflict is then detected as resolved. Reapplying (e.g: via automated rename) is much easier than what people typically mean by "manually resolving the conflict".

Also, as a pretty big productivity boost, it prints the conflicts in a way that lets many editors (sublime, emacs, etc) directly jump to conflicts, do the "git add" for you, etc. This converts your everyday editor into a powerful conflict resolution tool. Using the editing capabilities in most merge tools is tedious.

Re:The largest Git repo by flamedoge on Wednesday May 24, 2017 @04:19PM (#14413122)
or worse, formatting changes
Re:The largest Git repo by vtbassmatt on Wednesday May 24, 2017 @03:10PM (#14412628)
Windows developed an extension that lets them do conflict resolution in the web. We have a server-side API that it calls into, but the extension isn't fundamentally different from using BeyondCompare or $YOUR_FAVORTE_MERGETOOL.
Re:The largest Git repo by Flenser on Thursday May 25, 2017 @04:23AM (#14416017)
An extension to what? Could you open source it?
Re:The largest Git repo by vtbassmatt on Thursday May 25, 2017 @09:36AM (#14417162)
Extension to VSTS [1], sorry. We're working with them on making the extension available to the public. It's possible we could open source it as well; I'll poke around.

[1] https://www.visualstudio.com/en-us/docs/integrate/extensions...

Re:The largest Git repo by Luuseens on Thursday May 25, 2017 @11:07AM (#14417761)
Is this CodeFlow you're talking about?
Re:The largest Git repo by malnourish on Wednesday May 24, 2017 @02:40PM (#14412371)
For 3 way merging, I've had good luck with beyondcompare
Re:The largest Git repo by MikeHolman on Wednesday May 24, 2017 @04:31PM (#14413205)
Generally, it's the responsibility of whoever made the changes to resolve conflicts (basically, git blame conflicting lines and the people who changed them get notified to resolve conflicts in that file). Distributing the work like this makes the merges more reasonable.
Re:The largest Git repo by joeld42 on Wednesday May 24, 2017 @07:30PM (#14414265)
Yep, this is the way I work. Always rebase against master, and fix conflicts there, so branches merging into master should always be up-to-date and have zero conflicts.
Re:The largest Git repo by tokenizerrr on Wednesday May 24, 2017 @05:37PM (#14413664)
How does that work?
Re:The largest Git repo by srathi on Wednesday May 24, 2017 @08:36PM (#14414536)
You first rebase against the master locally and push the merged feature branch after resolving all the conflicts yourself. Afterwards, you go to the master and merge it against the updated feature branch. The 2nd merge should not result in any conflicts.
Re:The largest Git repo by nerdponx on Wednesday May 24, 2017 @10:16PM (#14414886)
This model can be annoying on running feature branches. Once you rebase, you have to force-push to the remote feature branch. It's not so bad if you use --force-with-lease to prevent blowing away work on the remote, but it still means a lot of rewriting history on anything other than one-off branches.
Re:The largest Git repo by cryptonector on Wednesday May 24, 2017 @11:53PM (#14415206)
No no, you never force push to "published" branches, e.g., upstream master. What you're doing when you rebase onto the latest upstream is this: you're making your local history the _same_ as the upstream, plus your commits as the latest commits, which means if you push that, then you're NOT rewriting the upstream's history.

(In the Sun model one does rewrite project branch history, but one also leaves behind tags, and downstream developers use the equivalent of git rebase --onto. But the true upstream never rewrites its history.)

Re:The largest Git repo by nerdponx on Thursday May 25, 2017 @12:41AM (#14415329)
That's what I thought as well. But what happens when you need to rebase the feature branch against master? Won't you have to force push that rebase?
Re:The largest Git repo by cormacrelf on Thursday May 25, 2017 @01:09AM (#14415401)
There's nothing stopping you from doing merges instead of rebases.

       * latest feature commit #3 (feature)
       * merge
      /|
     * | more master commits you wanted to include (master)
     | * feature commit #2
     | * merge
     |/|
     * | master commits you wanted to include
     | * feature commit #1
     |/
     *   original master tip
     *   master history...
Then, when you're done with feature, if you really care about clean history, just rebase the entire history of the feature branch into one or more commits based on the latest from master. I think checkout -b newbranch; rebase --squash master does the trick here:

     *   feature commits #1, #2 and #3 (newbranch)
     | * latest feature commit (feature)
     | * merge
     |/|
     * | more master commits you wanted to include (master)
     | * feature commit #2
     | * merge
     |/|
     * | master commits you wanted to include
     | * feature commit #1
     |/
     *   original master tip
     *   master history...
Then checkout master, rebase newbranch, test it out and if you're all good, delete or ignore the original.

     * feature commits #1, #2 and #3 (master, newbranch)
     * more master commits you wanted to include
     * master commits you wanted to include
     * original master tip
     * master history...
Re:The largest Git repo by eihakker on Tuesday May 30, 2017 @04:19PM (#14448220)

  *latest feature c
  *merge
 /|
* |more master comm. | feature commit #: | merge |/| * |master commits y | feature commit # |/ original master * master history ..
Re:The largest Git repo by cormacrelf on Wednesday May 31, 2017 @08:54PM (#14457120)
maybe try that again keeping 4 spaces before each line?
Re:The largest Git repo by cryptonector on Thursday May 25, 2017 @12:00PM (#14418112)
I've described this. Downstreams of the feature branch rebase from their previous feature branch merge base (a tag for which is left behind to make it easy to find it) --onto the new feature branch head.

E.g., here's what the feature branch goes through:

feature$ git tag feature_05

<time passes; some downstreams push to feature branch/remote>

feature$ git fetch origin feature$ git rebase origin/master feature$ git tag feature_06

And here's what a downstream of the feature branch goes through:

downstream$ git fetch feature_remote

<time passes; this downstream does not push in time for the feature branch's rebase>

downstream$ git rebase --onto feature_remote/feature_06 feature_remote/feature_05

Easy peasy. The key is to make it easy to find the previous merge base and then use git rebase --onto to rebase from the old merge base to the new merge base.

Everybody rebases all the time. Everybody except the true master -- that one [almost] never rebases (at Sun it would happen once in a blue moon).

Re:The largest Git repo by eridius on Wednesday May 24, 2017 @03:22PM (#14412719)
For 3-way merging the best tool I've found is Steve Losh's splice.vim (https://github.com/sjl/splice.vim/).
Re:The largest Git repo by babuskov on Wednesday May 24, 2017 @05:38PM (#14413670)
Re:The largest Git repo by 72deluxe on Thursday May 25, 2017 @05:06AM (#14416146)
Yes back when using Linux, I used Meld a lot. I can recommend it - the directory comparison is good.

Also KDiff3. Struggling to remember the other ones I used to try unfortunately.

Re:The largest Git repo by daemin on Thursday May 25, 2017 @10:15AM (#14417418)
I spent the better part of a year in a team that was merging and bug fixing our companies engine releases into the customer (and owning company) code base. We also had to deal with different code repositories and version control systems.

We ended up with a script that created a new git repository, checked out the base version of the code there. Then created a branch for our updated release and another for their codebase. Then attempted to do a merge between the two. For any files which couldn't be automatically merged it created a set of 4 files, the original then a triplet for doing a 3-way merge.

This is also when I bought myself a copy of Beyond Compare 4, which fits perfectly for the price and feature set for what we needed.

Re:The largest Git repo by InclinedPlane on Wednesday May 24, 2017 @09:50PM (#14414795)
It's usually not so bad. Generally a big project is actually split up into lots of different smaller projects, each of which have an owner. And typically a dev won't touch code that isn't theirs except in unusual cases. Teams that have more closely related or dependent code would typically try to work closer to each other and share code more often than teams that are more separated.
Re:The largest Git repo by cryptonector on Wednesday May 24, 2017 @08:45PM (#14414580)
Most changes don't touch hundreds/thousands of files. If you were to split the repo into many (as Windows used to be) then you'd still have the problem of huge projects having MANY conflicts, but worse: now you need to do your merge/rebase for each such repo.

In any case, rebasing is better than merging. (Rebasing is a series of merges, naturally, but still.)

Re:The largest Git repo by Flenser on Thursday May 25, 2017 @04:19AM (#14416008)
Are you aware of KDiff3[1]? If you are why do you prefer Xcode's opendiff?

[1] http://kdiff3.sourceforge.net/

Re:The largest Git repo by 72deluxe on Thursday May 25, 2017 @05:05AM (#14416141)
For me, I found I did not need to do 3 way merges that often and opendiff's native UI fits in better than kdiff3 (for me). I think kdiff3 was Qt? Despite Trolltech's best efforts, Qt does not feel native on a Mac.

This isn't to say that KDiff3 isn't great - it is.

Re:The largest Git repo by gnud on Thursday May 25, 2017 @05:30AM (#14416204)
I don't think KDiff 3 looks native anywhere, and I don't think it's because of Qt. It uses weird fonts and icons, and for some reason their toolbar buttons just look wrong.

Still an extremely useful program.

Re:The largest Git repo by aanm1988 on Wednesday May 24, 2017 @04:29PM (#14413192)
It's not nearly as bad when you have all the people who work on it with you.

You look at who caused conflicts and send out emails. Don't land the merge until all commits are resolved. People who don't resolve their changes get in trouble.

Not perfect, but there it is.

I worked on a chromium based product as well and had the exact same problem. Eventually we came up with a reasonable system for landing commits and just tried out best to build using chromium, but not having actual patches. Worked okay, not great. Better thant he old system of having people do it manually/just porting our changes to each release.

Re:The largest Git repo by draw_down on Wednesday May 24, 2017 @07:06PM (#14414137)
God, that sounds hellish.