Re:How release canaries can save your bacon
by solatic on Tuesday April 04, 2017 @01:45AM
(#14029824)
On the one hand, rollbacks need to be culturally normal. Finding out that an essential part of your process (essential because it keeps your mean-time-to-repair low) is undependable because the last time you ran it was a half a year so, and right now is precisely when you need it to be dependable, well, that royally sucks.
On the other hand, what you're talking about shouldn't be that hard to implement. Just hook your rollback system into your issue tracker to create a post-mortem issue whenever a rollback is necessary, and assign it to whoever initiated the deployment (or their manager). Easy.
On the third hand, you might end up finding that a lot of your post-mortems end up looking something like "we don't have reproduceable builds and we made a managerial decision not to invest in that now". And now you just created a ton of recurring paperwork for everyone with little benefit.
On the other hand, what you're talking about shouldn't be that hard to implement. Just hook your rollback system into your issue tracker to create a post-mortem issue whenever a rollback is necessary, and assign it to whoever initiated the deployment (or their manager). Easy.
On the third hand, you might end up finding that a lot of your post-mortems end up looking something like "we don't have reproduceable builds and we made a managerial decision not to invest in that now". And now you just created a ton of recurring paperwork for everyone with little benefit.