News for hackers. Stuff that matters.
Re:How release canaries can save your bacon by londons_explore on Tuesday April 04, 2017 @06:12AM (#14031042)
At our place, we rollback every few weeks just to test the system, even if nothing appears abnormal.

Next we plan to automate the process - 1 in 10 rollouts will actually be a rollout, a rollback, and another rollout, checking system health at each step.

Re:How release canaries can save your bacon by hectormalot on Tuesday April 04, 2017 @07:20AM (#14031411)
I just wanted to suggest this. Similar to 'chaos monkey' killing my processes once every few days, developers would also look differently at rollback procedures if I guaranteed that one in 10 would be rolled back randomly.