News for hackers. Stuff that matters.
Re:Improving how we deploy GitHub by cytzol on Monday January 25, 2021 @03:09PM (#25908096)
Something I found surprising is that a change to the GitHub codebase will be run in canary, get deployed to production, and then merged. I would have expected the PR to be merged first before it gets served to the public, so even if you have to `git revert` and undeploy it, you still have a record of every version that was seen by actual users, even momentarily.

Does anyone know the pros and cons of GitHub's approach?

Re:Improving how we deploy GitHub by bswinnerton on Monday January 25, 2021 @03:37PM (#25908454)
This is known as “GitHub Flow” (https://guides.github.com/introduction/flow/). I was pretty surprised by it when I first joined GitHub but I’ve grown to love it. It makes rolling back changes much faster than having to open up a revert branch, get it approved, and deploy it. When something goes sideways, just deploy master / main, which is meant to always be in a safe state.
Re:Improving how we deploy GitHub by samc98 on Monday January 25, 2021 @05:58PM (#25910109)
How does this work if two (or more) engineers touch the same file at the same time?

EG: If two engineers have two different PRs open modifying `index.html`, how does that go to canary?

Re:Improving how we deploy GitHub by bswinnerton on Monday January 25, 2021 @07:23PM (#25910826)
Before every deploy your branch has master merged into it. There’s some clever work by Hubot while you’re in line to deploy to create a version of your branch that has the potential new master / main branch. If conflicts arise you fix them before it’s your turn to deploy.
Re:Improving how we deploy GitHub by rtpg on Tuesday January 26, 2021 @03:41AM (#25913726)
how deep does this deploy queue get? And does everything that lands on master have to first be deployed?

This seems like a bit tough for getting work done if you have enough people all waiting around for this

Re:Improving how we deploy GitHub by jules2689 on Tuesday January 26, 2021 @10:43AM (#25917068)
The deploy queue usually gets to be a couple of "trains" deep which usually includes work from 4-10 devs. This represents a couple of hours. We have had issues with it taking too long, but this work I wrote about has improved that! We continue to try to improve it.
Re:Improving how we deploy GitHub by gowld on Tuesday January 26, 2021 @02:27PM (#25920058)
Why would you revert a branch (in source) before deploying it? You already have the old known good version in the history, so you should deploy that.
Re:Improving how we deploy GitHub by halukakin on Monday January 25, 2021 @03:37PM (#25908449)
I think this method seems to get more popular by day. IMHO, previously master was the branch you merge before the deploy process. But today this is reversed.

The main benefit is, other developers can rely on the master branch even more. They will know there will not be a revert on the master branch they just pulled one hour ago and already started coding on.

Re:Improving how we deploy GitHub by Kwpolska on Monday January 25, 2021 @05:05PM (#25909575)
A `git revert` creates a new commit. To a developer, a revert commit appearing on master has the same effect as a pull request (or ten) being merged into it. If the revert affects code you’re working on, you will need to resolve conflicts, just like you would need to if a merged PR affected the same code.
Re:Improving how we deploy GitHub by irjustin on Monday January 25, 2021 @09:13PM (#25911629)
While what you're saying is true for all situations, the of the github flow is that to a developer relying on master, problematic code never made it in.

Agreed that any codes can be added/removed but those are 100% valid changes in github flow.

Re:Improving how we deploy GitHub by lucasyvas on Tuesday January 26, 2021 @10:45AM (#25917101)
I thought this was what tests were for :)
Re:Improving how we deploy GitHub by cutemonster on Tuesday January 26, 2021 @07:59PM (#25922945)
Large software systems need all of tests, canaries, rollbacks.
Re:Improving how we deploy GitHub by trabant00 on Tuesday January 26, 2021 @02:17AM (#25913365)
I've worked with this approach for the past year: we have a "release" named branch, we deploy that, run all automated tests in production, wait a couple of hours for feedback including customer support, then merge it to master.

This means master is a history of proven stable builds. And in an emergency you don't have to think about what to roll back to, it's by default master.

It's just a convention, it does not have any real benefits or downsides. You could do the same with another branch named stable, or with tags.

> a record of every version that was seen by actual users

This is covered by the release branches and also by the CD pipeline.

Re:Improving how we deploy GitHub by konschubert on Tuesday January 26, 2021 @04:46AM (#25914053)
This only works if you have less than one release per day
Re:Improving how we deploy GitHub by gowld on Tuesday January 26, 2021 @02:28PM (#25920066)
Why?
Re:Improving how we deploy GitHub by cutemonster on Tuesday January 26, 2021 @08:03PM (#25922981)
> wait a couple of hours for feedback including customer support, then merge it to master

Seems one can deploy up to ... Say, 24/3? times per day? If 3 h in between

(trabant00 thanks for explaining :-))

Re:Improving how we deploy GitHub by majormajor on Monday January 25, 2021 @09:15PM (#25911649)
> I would have expected the PR to be merged first before it gets served to the public, so even if you have to `git revert` and undeploy it, you still have a record of every version that was seen by actual users, even momentarily.

This sounds rather terrifying to me as well. Hopefully there's some sort of system for keeping around all those individual branches that made it out to prod, however briefly, for future debugging/auditing purposes if you ever need them.

It's never fun to have to be doing "what code was running at this time" investigations, but every once in a while it's the only way to really get to the root of something.

Re:Improving how we deploy GitHub by Existenceblinks on Tuesday January 26, 2021 @11:47AM (#25917984)
My ex-company has similar flow, but fix the terrifying part by (off-git-hosting, i.e. in-memory) merging ALL opened PRs to master and deploying straight to staging server. Any PR can be marked as excluded from a deploy. All PRs are ALWAYS based off master. tldr; master based pull requests as source of release.
Re:Improving how we deploy GitHub by cies on Tuesday January 26, 2021 @04:37AM (#25913989)
We also make a release branch (releases/x.y.z), tag a release candidate in it (x.y.z-RCn), build it, deploy it, wait for a bit (canary stage?), it we did not want to rollback: then merge it into master.