wvanbergen

This is the value of consulting. If it works, you get to claim the win. If it fails, you blame the consultant.

Sounds like a similar setup to GAIA on Galiano Island in BC, described here: https://www.galianonet.org/about

> Being as we use different tools, we wouldn’t allow a pull to be merged if it wasn’t up-to-date with master which is similar but a different approach

That wouldn't work for us due to the amount of changes we need to ship. If you rebase your branch and wait for CI to come back green, chances are another PR will have merged in the mean time, which means your rebased branch is no longer up to date with master. You end up stuck in a rebase cycle.

For this reason, we have no choice but to batch PRs, which is what the merge queue tool does. Faster CI will reduce this problem and we're working on that as well, but won't fully solve this.

> I guess I’m not fully understanding how a queue prevents this. Since you don’t have a full picture of the state of master until something is merged from the queue, how do the CI checks in the queue prevent things that branch-based CI checks wouldn’t prevent in a “develop” branch? With branches and develop, pull requests remain open until they can be assured they merge properly with develop as well.

The trick of the merge queue is that it splits the "merging a branch / pull request" in two steps:

1. Create a merge commit with master and your PR branch as ancestors.

2. Update the `master` ref to point to the merge commit.

Normally when you press the "Merge Pull Request" button, it will do those two things in one go. By splitting it up in two distinct steps, we can run CI between step 1 and 2, and only fast-forward master if CI is green.

This means that master only ever gets forwarded to green commits. And because the sha doesn't change during a fast-forward, all the CI statuses are retained. Only when we fast-forward will GitHub consider to pull request merged, so we don't have to "undo" pull request merges when they fail to integrate. If the merge commit fails to build successfully, we leave a comment on the PR that merging failed, and the PR is still open.

When we have multiple PRs in the queue, we can create merge commit on top of merge commit, and run CI on those merge commits. When once of these CI runs comes back, we can fast forward master to it, and potentially merge multiple pull requests at once with this approach.

Generally our metrics and exception reports are tagged with the sha and the deploy stage.

We generally require 2 reviewers, and no sign-off on deploys. For PCI-compliant code things work a bit differently, but tries to follow this as closely as possible.

A `develop` branch has several disadvantages.

You will want to make your `develop` branch the default branch in git and on GitHub, to make sure pull requests automatically are targeted properly (not doing this would be a major UX pain). However, that means that when you `git clone` a repository you are not guaranteed to get a working version.

The `develop` branch can still be broken, which is a problem that needs to be addressed. While you can revert breaking changes (or force-push it to a previous known good sha), and you can automate this process, the pull request is already marked as merged at this point. This means that developers have to open a new PR whenever that happens.

With the queue approach, pull requests remain open until we are sure they integrate properly. Also, we have the opportunity to use multiple branches to test different permutations of PRs, so we can still progress and merge some PRs even if the "happy path" that includes all PRs does not integrate properly.

The branches that are being merged are tested, also in the first version. However, different branches can conflict with each other, and break the master builds. This was happening often enough for us to want to prevent this.

The simple way would be to rebase your branch (or merge in master). However, with the amount of changes that are being merged, by the time the CI result comes in for your rebased branch, the tip of master is already changed making the CI result obsolete. So we went for the queue approach instead.

Start by reading this: http://wiki.c2.com/?WhyIsPayrollHard

As a Dutch immigrant to Canada and a Shopify employee, I can assure you that the Germany-Netherlands rivalry is alive and well at this side of the ocean :)

All kidding aside, I cannot upvote this enough. The difference in how immigrated is treated is vast between Canada and the Netherlands. Yes, people complain about it here too, but success stories like this and a history of immigration makes sure the general opinion on immigration is favourable.

We actually evaluated other options as well, including Angular, but found them not the best solution for our problem. We documented our experiments and evaluations at the time here: https://gist.github.com/kristianpd/f4c2e0aeb53d09f6def1

This. We ended up creating models for the exact same stuff in both Ruby and JS. Lots of code duplication, and losing client-side performance due to deserializing JSON all the time.

fidz: should be fixed now. Thanks for reporting this.

Ah, this is a bug in our redirect system. I will look into it.

MattyMc: sure. Primarily: be yourself, show what you are passionate about, and be willing to adopt change. When applying: cover letter > resumé, and try to stand out because we get many applications. Email me at willem at shopify dot com if you have any specific questions.

At Shopify, we have moved to Docker for deploying our main product. Primary advantages for us are faster deploys, because we can do part of the old deploy process as part of the container build. Secondly: easier scalability, because we can add additional containers to have more app servers or job workers. More info at http://www.shopify.com/technology/15563928-building-an-inter...

Textmate.

My point is not so much that REST is wrong, but that RESTful thinking can be harmful if it comes before thinking about the process you're supporting.

Must have disappeared when I transferred the article. Fixed, and thanks for reporting.

Thanks, will fix.

My post isn't meant as a description about state machines; I point to wikipedia for that. The Ruby gem also gives a very nice introduction indeed.

However, the point I try to make is very much about state machines, because modelling state using a state machine, forces you to think about all possible state transitions. This will make it less likely that you application will get into an unexpected state. Also, keeping an audit trail is very much about state transitions, because the transitions describe the behavior of the system, not the state itself.

Indeed. If you do not use a state machine in a situation like that, you either get very complicated code as well, or you conveniently do not implement some of the state transitions. That will always get back to you: "how can this ever happen?!"

State machines force you to consider all possible state transitions. This makes sure that transitions are handled properly, or they are actively made impossible.

My favorite issue in 1.9.2: "." is no longer included on the $LOAD_PATH.