News for hackers. Stuff that matters.
Re:Ask HN: How do you roll back production? by karka91 on Sunday November 03, 2019 @03:20PM (#21436309)
Assuming this is about web dev.

Nowadays - flip a toggle in the admin. Deployments and releases are separated.

Made a major blunder? In kubernetes world we do "helm rollback". Takes seconds. This allows for a super fast pipeline and a team of 6 devs pushes out like 50 deployments a day.

Pre-kubernetes it would be AWS pipeline that would startup servers with old commits. We'd catch most of the stuff in blue/green phase, though. Same team, maybe 10 deployments a day but I think this was still pretty good for a monolith.

Pre-aws we used deployment tools like capistrano. Most of the tools in this category have multiple releases on the servers and a symlink to the live one. If you make mistake - run a command to delete the symlink, ln -s old release, restart web server. Even though this is the fastest rollback of the bunch the ecosystem was still young and we'd do 0-2 releases a day.

Re:Ask HN: How do you roll back production? by Shalle135 on Sunday November 03, 2019 @04:13PM (#21436641)
Why not canary releases? You can load balance for example 1% of the traffic to the new deployment and see if you experience any issues. If you do - you just change the loadbalancer to use the known good pods.
Re:Ask HN: How do you roll back production? by karka91 on Monday November 04, 2019 @04:44AM (#21440142)
Honestly, we haven't considered that. Breakage rarely reaches production, rollbacks are quick, a process orchestrator retries most of the stuff and because of that we do not have a problem worth solving with "canary deployments". Note that we do sometimes carry out "canary releases" via toggles.
Re:Ask HN: How do you roll back production? by pojzon on Sunday November 03, 2019 @05:28PM (#21437163)
How you take care of DB updates when using cannary deployments ? For example those which are not backwards compatible ?

Ps. Releases are about building new versions of code packages. Deployments about pushing them out to environments.

Re:Ask HN: How do you roll back production? by karka91 on Monday November 04, 2019 @04:59AM (#21440213)
This depends a lot on the databases used and flexibility of the code that's accessing the data. One method is to deploy in multiple stages and to use views. In pre-deploy you create a view with schema/data needed for the release. In deploy stage you roll-out the canary code. In post-deploy stage remove either the old data/schema on success or new data/schema on failure. It's quite an overhead to implement and maintain this process.
Re:Ask HN: How do you roll back production? by bpicolo on Sunday November 03, 2019 @06:37PM (#21437571)
Best way is to not write forwards / backwards incompatible database changes.
Re:Ask HN: How do you roll back production? by korijn on Sunday November 03, 2019 @04:15PM (#21436660)
> Nowadays - flip a toggle in the admin. Deployments and releases are separated.

Would you mind explaining this a little further? How does the separation allow you to flip a switch in the admin?

Re:Ask HN: How do you roll back production? by rgoulter on Sunday November 03, 2019 @08:29PM (#21438197)
AFAIU this is making use of feature flags.

Uploading new code to a server where the new-code is behind a disabled feature flag means the program hasn't changed. The feature flag can be enabled when it's suitable. (This could even be 'rolled out' to only a subset of users; e.g. testers, internal, 10% of users, etc.)

deploy = put the code on the server, release = enable the feature flags