Re:A terrible, horrible, no-good, very bad day at Slack
by cutemonster on Tuesday July 07, 2020 @03:13PM
(#23762922)
What is, according to you, and the SRE book, a good time during the day/week/year to do these real test outages? How much downtime could be ok?
For, let's say, a b2b saas
They notify the customers first? Like, "we'll a little bit sabotage our, well, your, servers this weekend, to find out if they fail and shutdown completely and cannot start again" :-)
Re:A terrible, horrible, no-good, very bad day at Slack
by peterwwillis on Wednesday July 08, 2020 @12:13AM
(#23766829)
It doesn't have to be a customer impact to bring down a service or parts of a service (if it's designed correctly). But even if it was a customer impact, you can do it at the same time you'd schedule regular maintenance (there's always something that needs maintenance, eventually) and throw up a maintenance page.
It's up to the business to define SLAs and SLOs that the customer will be satisfied with.
Re:A terrible, horrible, no-good, very bad day at Slack
by cutemonster on Wednesday July 08, 2020 @05:12AM
(#23768223)
> do it at the same time you'd schedule regular maintenance ... SLA ... SLO
I'll do that (some day), thanks
In that process you would go, "Ok, the service is failed and I don't know why. How does the monitoring look?", and then you'd notice it was broken. This is best performed by the person on the team with the least experience, so they ask the most questions, which reveals more.
What's really funny is when the recovery you expected to work doesn't work, and then you have bigger problems... But at least you planned for it :)