News for hackers. Stuff that matters.
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by tomcam on Friday October 27, 2023 @02:31PM (#38042138)
> But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system

Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by shadowgovt on Friday October 27, 2023 @03:37PM (#38042922)
Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt.

But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."