Ask HN: How do you roll back production?

Re:Ask HN: How do you roll back production? by rvdmei on Sunday November 03, 2019 @11:26AM (#21434777)

Rolling back is usually a bad practice and can get quite challenging if not impossible in distributed environments.

If you can pinpoint a specific commit that is causing the issue. Revert that commit and go through your standard release process.

[ Parent | on HN ]

Re:Ask HN: How do you roll back production? by mooreds on Sunday November 03, 2019 @11:41AM (#21434886)

Would love to know more about how you roll forward in that case. Or do you canary so thoroughly that bugs never get to prod?

[ Parent | on HN ]

Re:Ask HN: How do you roll back production? by rvdmei on Sunday November 03, 2019 @12:03PM (#21435050)

Bugs do get to prod, that’s just reality. If you keep changes small and release often you won’t see many big bugs in production. We typically keep master in an “always deployable” state. If a bug gets found in prod, we either fix it or create a new commit that reverts the commit that caused the problem. If it’s a faulty migration, reverting the commit usually won’t work and we have to fix the bug. For QA we use a mix of automated tests and manual tests.

[ Parent | on HN ]