News for hackers. Stuff that matters.
Re:Ask HN: How do you roll back production? by gfodor on Sunday November 03, 2019 @11:48AM (#21434950)
Yup, can't beat blue/green. A big part of the reason I recommend it is because unlike most other rollback schemes, the actual mechanism for rolling back is the same as for a normal release (a load balancer flip), so it's continually exercised and validated. To roll back, you literally do a deploy, but you just skip the step where you alter the bits on the dark cluster. This is highly unlikely to fail, since the only thing that has to not go wrong is the part where the software update is skipped, which basically boils down to a conditional.

Any mechanism for rollbacks that isn't tested continuously is likely to fail during incident response. It's a huge anti-pattern to have 'dark' processes only used during incident response -- same thinking behind why you should also be continually testing your backups, continuously killing servers to verify recovery, etc.

Re:Ask HN: How do you roll back production? by Angostura on Sunday November 03, 2019 @12:00PM (#21435029)
They you just have to worry about rolling back the changed database schema that you have rolled out to both green and bluie variants - I presume. That bit feels tricky
Re:Ask HN: How do you roll back production? by gfodor on Sunday November 03, 2019 @12:03PM (#21435052)
Yep, that's a fair point. In scenarios where you have made non-backwards compatible database changes, this approach cannot be used. Typically the dark cluster is accessible for pre-flight testing, so you can run tests there to determine if the service is still operating as expected despite the database being one migration step ahead. In your response runbook, you should have a contingency for the scenarios where that happens. (For many projects, the % of changes that have this characteristic are small enough that it doesn't move the needle on the risk profile, but Murphy's law can mess with you :))