News for hackers. Stuff that matters.

A comment on Prime Down: Amazon’s sale day turns into fail day. Parent comment

Re:Prime Down: Amazon’s sale day turns into fail day by snewman on Monday July 16, 2018 @05:23PM (#17545260)
Yup, many an outage has something like this at its core. At my company, to address that, we've built an internal library for enforcing rate limits and size limits that is (a) configurable on the fly and (b) generates logs, so that we can trigger an alert whenever any limit reaches 70% of capacity. And thus, hopefully, head these things off before they reach the tipping point.

https://blog.scalyr.com/2014/08/99-99-uptime-9-5-schedule/

[ Parent | on HN ]
Re:Prime Down: Amazon’s sale day turns into fail day by codebje on Monday July 16, 2018 @08:40PM (#17546400)
Don't forget to alert when the double derivative of capacity growth is above a (very low!) threshold, that can catch an explosive problem far earlier than the 70% mark, by which time it might be growing at 5% per second.
[ Parent | on HN ]
Re:Prime Down: Amazon’s sale day turns into fail day by cheeze on Monday July 16, 2018 @10:07PM (#17546761)
It might also be a momentary blip that you really shouldn't need to wake a human up for. The hard part of monitoring imo is that perfect balance. Asymtotically approaching nirvana but never quite reaching.
[ Parent | on HN ]
Re:Prime Down: Amazon’s sale day turns into fail day by codebje on Tuesday July 17, 2018 @01:39AM (#17547541)
All monitoring has that problem :-) But if your rate of rate of change has been positive for some suitable amount of time for the context, it's worth waking a human up over it, because the amount of resource remaining is dwindling exponentially.
[ Parent | on HN ]
Re:Prime Down: Amazon’s sale day turns into fail day by bostonvaulter2 on Monday July 16, 2018 @09:01PM (#17546494)
That sounds like a very interesting metric to track and report on. Do you have any further references that discuss that approach?
[ Parent | on HN ]
Re:Prime Down: Amazon’s sale day turns into fail day by codebje on Tuesday July 17, 2018 @01:37AM (#17547534)
Not offhand, sorry - I first encountered the notion at a conference some years back, but I've long since gotten out of operations so it's all a bit dusty for me now.
[ Parent | on HN ]
Re:Prime Down: Amazon’s sale day turns into fail day by adreamingsoul on Monday July 16, 2018 @05:40PM (#17545396)
That's cool, but it's a bit more complicated then rate limits.
[ Parent | on HN ]