News for hackers. Stuff that matters.
Re:Ask HN: How do you roll back production? by Shalle135 on Sunday November 03, 2019 @04:13PM (#21436641)
Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.
Re:Ask HN: How do you roll back production? by karka91 on Monday November 04, 2019 @04:44AM (#21440142)
Honestly, we haven't considered that. Breakage rarely reaches production, rollbacks are quick, a process orchestrator retries most of the stuff and because of that we do not have a problem worth solving with "canary deployments". Note that we do sometimes carry out "canary releases" via toggles.
Re:Ask HN: How do you roll back production? by pojzon on Sunday November 03, 2019 @05:28PM (#21437163)
How you take care of DB updates when using cannary deployments ? For example those which are not backwards compatible ?

Ps. Releases are about building new versions of code packages. Deployments about pushing them out to environments.

Re:Ask HN: How do you roll back production? by karka91 on Monday November 04, 2019 @04:59AM (#21440213)
This depends a lot on the databases used and flexibility of the code that's accessing the data. One method is to deploy in multiple stages and to use views. In pre-deploy you create a view with schema/data needed for the release. In deploy stage you roll-out the canary code. In post-deploy stage remove either the old data/schema on success or new data/schema on failure. It's quite an overhead to implement and maintain this process.
Re:Ask HN: How do you roll back production? by bpicolo on Sunday November 03, 2019 @06:37PM (#21437571)
Best way is to not write forwards / backwards incompatible database changes.