News for hackers. Stuff that matters.
Re:Linux Crisis Tools by Spivak on Sunday March 24, 2024 @12:50AM (#39805227)
Y'all don't do post-mortem investigations / action items?

I get the desire to troubleshoot but priority 0 is make the system functional for users again, literally everything else can wait. I once had to deal with an outage that required we kill all our app servers every 20 minutes (staggered of course) because of a memory leak while it was being investigated.

Re:Linux Crisis Tools by Salgat on Sunday March 24, 2024 @01:29AM (#39805331)
Usually depends on the impact. If it's one of many instances behind a load balancer and was easily fixed with no obvious causes, then we move on. If it happens again, we have a known short-term fix and now we have a justified reason to devote man-hours to investigating and doing a post-mortem.
Re:Linux Crisis Tools by kqr on Sunday March 24, 2024 @07:05AM (#39806398)
> I get the desire to troubleshoot but priority 0 is make the system functional for users again, literally everything else can wait.

What numbers went into this calculation, to get such an extreme result as concluding that getting it up again is always the first priority?

When I tried to estimate the cost and benefit, I have been surprised to make the opposite conclusion multiple times. We ended up essentially in the situation of "Yeah, sure, you can reproduce the outage in production. Learn as much as you possibly can and restore service after an hour."

This is in fact the reason I prefer to keep some margin in the SLO budget -- it makes it easier to allow troubleshooting an outage in the hot path, and it frontloads some of that difficult decision.