Re:Former Japan PM Shinzo Abe Unconscious After Shooting; Ma...
by mjr00 on Friday July 08, 2022 @12:44AM
(#32023291)
There are both models at Amazon. Individual teams have their own on-call; if for example an RDS instance is continually failing to boot, someone on-call (edit: from the RDS team) will get paged to look at it.
I believe order drops are looked at by dedicated reliability team, because it's such a significant problem. But they don't watch graphs because... why would they? They set up the machines to page them when metrics aren't within an expected range. Which is what happened.
Re:Former Japan PM Shinzo Abe Unconscious After Shooting; Ma...
by majormajor on Friday July 08, 2022 @12:48AM
(#32023321)
That makes a lot of sense.
The ops teams I've worked with watched graphs because (a) sometimes you see interesting changes or trends before they turn into actionable alerts and people can look into them/figure them out before shit's on fire, and (b) it makes it look more like you're doing something when everything is smooth sailing and not actively on fire, solving the "wait why are we paying for this if it doesn't take us long to respond to incidents anyway" naive-meddlesome-exec problem.
I don't know which model Amazon uses myself, but it sounds more like the "everyone is their own first responder" model which surprises me a little, but each model has its pros and cons. The justification I'm most familiar with is "if there's a separate ops team you're going to be complacent and throw shit over the wall without caring about quality" which is a somewhat lazy excuse for not actually getting your engineers to care about quality with anything but a hammer to beat them on the head with, but ....