Re:GitHub incident 2022-03-23
by raffraffraff on Wednesday March 23, 2022 @03:10PM
(#30782093)
Yuck. Honestly, restarting a database to fix a major outage sounds like "we have no idea what we're doing"
[ Parent
| on HN ]
Re:GitHub incident 2022-03-23
by blueplanet200 on Wednesday March 23, 2022 @07:45PM
(#30785004)
It sounds like "they don't know why it's going down." I've worked with plenty super competent people that have taken time to root cause incidents.
Guide to incidents: Step 1: Stop the bleeding Step 2: Prevent it in the future
Doing Step 1 doesn't make you incompetent.
Re:GitHub incident 2022-03-23
by raffraffraff on Thursday March 24, 2022 @01:05PM
(#30792602)
I'm not a DBA, and maybe you're not a DBA either, so this question goes to DBAs who may be reading: aren't you always better off killing the bad queries instead of rebooting the whole box, if that's an option? (ie: aside from times when the entire host is screwed, load per core is >50, metrics aren't getting out, you can't ssh in etc)
Sporadic database performance issues can certainly make you feel that way. They are definitely not trivially debugged at scale
Would you rather it stay down while they spend a day debugging it?
Re:GitHub incident 2022-03-23
by paulryanrogers on Wednesday March 23, 2022 @05:47PM
(#30783886)
If that means it won't be down every morning in my time zone then yes.
As long as it's announced in advance so that users/customers can plan ahead, I don't see why not.