News for hackers. Stuff that matters.

A comment on Lessons Learned from Twenty Years of Site Reliability Engineering. Parent comment

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by shadowgovt on Friday October 27, 2023 @12:33PM (#38040651)
There was a famous incident at one point where code search internally went down. It turned out that while they had deployed the tool internally, one piece of the indexing process was still running as a cron job on the original developer's desktop machine. He went on vacation, his credentials aged out, and of crawler stopped refreshing the index.

But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system and discovered that the key needed to restore the password vault system was stored in s aafe, the combination for which had been moved into the password vault system. Only with advanced, modern technology can you lock the keys to the safe in the safe itself with so many steps!

[ Parent | on HN ]
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by tomcam on Friday October 27, 2023 @02:31PM (#38042138)
> But my favorite incident will forever be the time they had to drill out a safe because they were disaster-testing the password vault system

Great story to be sure—-but I’m going to call it a success. They did the end-to-end testing and caught it then instead of real life.

[ Parent | on HN ]
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by shadowgovt on Friday October 27, 2023 @03:37PM (#38042922)
Well, mostly-kinda-sorta. ;) It's the internal password vault, and there's only one of them, so it's more like "they broke it on purpose and then had to fix it before the company went off the rails." Among the things kept in that vault are credentials that if they age out or aren't regularly refreshed, key internal infrastructure starts grinding to a halt.

But still, "it broke while engineers were staring at it and trying to break it" is a better scenario than "it broke surprisingly while engineers were trying to do something else."

[ Parent | on HN ]