Re:Lessons Learned from Twenty Years of Site Reliability Eng...
by shadowgovt on Friday October 27, 2023 @03:37PM
(#38042922)
Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt.
But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."
Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.