Re:Show HN: Calculate how long your website/API can be down
by mcharezinski on Saturday May 30, 2020 @09:27PM
(#23366422)
Shooting ourselves in the foot was the number one cause of downtime.
What do you mean?
Re:Show HN: Calculate how long your website/API can be down
by jedberg on Saturday May 30, 2020 @10:23PM
(#23366647)
The biggest cause of downtime was deployment. Either deploying code with a bug that wasn't caught in testing, or changing a real-time configuration parameter that wasn't properly scoped.
As opposed to a scaling issue that showed up later, or a node failure or all the other things that could cause downtime.
We ended up settling on predicting how much traffic there should be at any given second, and then going from there. For example, if there should have been 3000 stream starts in that second, and we got 2000 starts, that would count as 1/3 of a second of downtime.
Our goal was 4 nines, which allowed for a total of one minute of downtime a week. We managed to achieve that on many weeks of the year. One week in the three years I kept close track we managed 100%. It was near Christmas, when no one was working and deploying code. Shooting ourselves in the foot was the number one cause of downtime.