If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken. Then you can do the fix again, but with more care and testing. If all goes well, you only have to think about this issue for 10 minutes, and then get on with your life, you don't have to spend 10 minutes a day following it through the 2 week deployment process and only then notify the customer that they can finally use the product.

Also, if you think a fix needs more care, the ability to ship quickly doesn't impede doing it slowly; but in many companies there is simply no ability to ship fast, regardless of the type of bug.

> If it only takes 1 minute to push a fix, it usually will take about the same amount of time to revert it, if it's broken.

Unless it caused data corruption.

Yes, that's why I said usually. It's also why I said, you can take it slow if you're concerned. And something that would possibly cause data corruption is a good thing to take more time with.

In a perfect scenario you would have set up continuous integration that automatically runs all your unit, integration and end-to-end tests. Ideally your coverage with tests is so good that you can rely on a successful integration of the bug fix commit.

This of course is a ideal scenario that you have to work on seriously to be very close to it.

Continuous deployment works differently. Your approach focuses on reducing MTBF; continuous deployment focused on reducing MTTR.

For CD to work, you can't really depend on QA for regression testing. You need to have really good automated test suites. You still do want exploratory QA, but that doesn't have to happen before release.

Code review works similarly. As your quote says, he did proper code review. It was a very small change, so it doesn't take long. Strong test coverage means code review is less about finding possible bugs and more about design issues. There's no urgency to getting those before release; people can watch changelogs just as well as they can service a review queue. And with frequent releases, developers return to important code often, meaning that design improvements happen over time.

You deploy gradually, initially to 1% or so of requests. If your metrics, KPIs, etc. don’t suffer, it continues to roll-out until 100% of requests are using that code-path - within a few hours or so.

Takes a bit of infra effort, but for a company like Intercom, that’s very easily achievable.

this is easy and doable if your change is a front end change.

good luck trying this when its a backend change involving distributed threads, locks, resources, etc where production load is 100x more than staging/test.

If the cost of downtime is high enough you can mirror some or all traffic onto a shadow copy of the infrastructure.

Obviously in those cases, you’d take more than a minute to review prior to deployment...

There are plenty of backend changes where your list of complexities do not apply.

what would be some good resources to learn how to build such infrastructure? I've seen it as a feature of various platforms but would imagine it could become complex otherwise.

It wouldn’t be practical without some kind of underlying infrastructure orchestration system. Obviously some companies have gone the extra step to actually build these systems - e.g. Google made Kubernetes. For any not-huge companies, I think it pretty much just falls to choosing one of the many existing tools.

Obviously it is possible to just throw together a bunch of scripts and vaguely make it all happen, I’m not sure I’d be comfortable to do that in production.

The simple way to do this can apply if you're mostly doing things in the context of a user. Write the change to check if a hash of the user identifier is less than a target; increase the target over time.

Yes, because otherwise non-critical bugs pile up and hide critical bugs.

Another solution to this is regular bug triage.

Stale inventory is expensive.