Re:Ask HN: Has anyone migrated off containerized infrastruct...
by allannienhuis on Friday August 14, 2020 @10:59AM
(#24158761)
Pair programming anything as urgent as a hotfix works really well. It takes some pressure off of the developer working on it and turns it into a team event.
We will even sometimes keep the video call open until deployment is done and production validation is complete - the devops guys get the information they need, someone else is keeping an eye on the checklist and calling out items if necessary, etc.
Re:Ask HN: Has anyone migrated off containerized infrastruct...
by mumblemumble on Friday August 14, 2020 @11:57AM
(#24159595)
Oh yeah, absolutely. I was happiest when I also had the attending ops person looking over my shoulder while I worked on the fix.
Re:Ask HN: Has anyone migrated off containerized infrastruct...
by codemonkey-zeta on Friday August 14, 2020 @08:58AM
(#24157290)
I really like that approach. It also takes pressure off the tech lead or whoever is implementing the fix, and transforms the patch into a full-team responsibility. I bet this sort of behavior makes for strong and effective teams.
I worked at a trading firm where you could add a zero to that cost for a 3-hour outage, and I can tell you that the one thing we would absolutely never skimp on was code review. Because the cost of a bad "fix" that actually makes things worse has the potential to be greater still, and because humans are most likely to make silly mistakes when they're working under intense pressure.
What we would do instead is slightly intensify the code review process by pair programming the hotfix, and ensuring that a third developer who was familiar with the system in question was standing by to follow up with an immediate review.