Re:The Mystery Blips
by retrocryptid on Saturday November 19, 2022 @10:28AM
(#33670975)
Lol. During "high velocity events" at Amazon, we would have a war room with all the engineers and a couple old hands directing responses.
Some of our metrics came in 5 minutes delayed which wasn't a problem for normal days.
These metrics moved slow enough that when you got an alarm, there was still plenty of time to take corrective action.
But for HVEs this was an issue. During black Friday or prime day, sometime some metrics spiked so fast you had no time to respond (usually from people hitting page reload a few minuted before a sale kicked off.)
To get an idea for what was going on, I would go in twitter and search for things like "amazon failure" or "amazon 502."
We often got reports of problems via Twitter before problems showed up on our dashboards.
Some of our metrics came in 5 minutes delayed which wasn't a problem for normal days. These metrics moved slow enough that when you got an alarm, there was still plenty of time to take corrective action.
But for HVEs this was an issue. During black Friday or prime day, sometime some metrics spiked so fast you had no time to respond (usually from people hitting page reload a few minuted before a sale kicked off.)
To get an idea for what was going on, I would go in twitter and search for things like "amazon failure" or "amazon 502."
We often got reports of problems via Twitter before problems showed up on our dashboards.