> The mistake of ignoring the maxage of the request queue is perhaps the second most common blind spot in every Ops team I've ever worked with. No insult to my fellow Ops folks, but we've got to stop overlooking this.
What's the first?
Re:Slack’s Outage on January 4th 2021
by floatingatoll on Monday February 01, 2021 @02:52PM
(#25992863)
Non-randomized wallclock integers.
For example: “sleep 60 seconds”, “cron 0 * * * * command”, “X-Retry-After: 300”
Found in: recurring jobs, backoff algorithms, oauth tokens.
Found in: ops-created tasks, dev-released software.
I'm building something at Cronitor to help detect those hot-spots! If you want to learn more, email me: shane at cronitor.io
Re:Slack’s Outage on January 4th 2021
by floatingatoll on Monday February 01, 2021 @08:44PM
(#25996578)
Tell us more here!
Slack wrote an autoscaling implementation that ignored request queue depth and downsized their cluster based on CPU usage alone, so while they knew how to resolve it, I would not go so far as to say they knew how to prevent it. The mistake of ignoring the maxage of the request queue is perhaps the second most common blind spot in every Ops team I've ever worked with. No insult to my fellow Ops folks, but we've got to stop overlooking this.