How release canaries can save your bacon

Re:How release canaries can save your bacon by ams6110 on Monday April 03, 2017 @09:53PM (#14028926)

From that link: At Google, our philosophy is that “rollbacks are normal.” When an error is found or reasonably suspected in a new release, the releasing team rolls back first and investigates the problem second.

I like that -- reminds me of aviation, where a go-around is normal. If your approach to landing isn't stabilized, you're too high, too low, too fast, too slow, etc. don't try to save it. Go around and try again.

[ Parent | on HN ]

Re:How release canaries can save your bacon by londons_explore on Tuesday April 04, 2017 @06:12AM (#14031042)

At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal.

Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.

[ Parent | on HN ]

Re:How release canaries can save your bacon by hectormalot on Tuesday April 04, 2017 @07:20AM (#14031411)

I just wanted to suggest this. Similar to 'chaos monkey' killing my processes once every few days, developers would also look differently at rollback procedures if I guaranteed that one in 10 would be rolled back randomly.

[ Parent | on HN ]