Re:Lessons Learned from Twenty Years of Site Reliability Eng...
by hinkley on Friday October 27, 2023 @05:23PM
(#38044156)
Before divestiture we had two groups splitting responsibility for IT, so our DevOps team (bleh) only had partial control of things. We were then running in heterogenous mode - one on prem data center and cloud services.
One day a production SAN went wonky, taking out our on prem data center. ... and also Atlassian with it. No Jira, no Confluence. Possibly no CI/CD as well. No carefully curated runbooks to recover. Just tribal knowledge.
People. Were. Furious. And rightfully so. The 'IT' team lost control of a bunch of things in that little incident. Who puts customer facing and infrastructure eggs into the same basket like that?
One day a production SAN went wonky, taking out our on prem data center. ... and also Atlassian with it. No Jira, no Confluence. Possibly no CI/CD as well. No carefully curated runbooks to recover. Just tribal knowledge.
People. Were. Furious. And rightfully so. The 'IT' team lost control of a bunch of things in that little incident. Who puts customer facing and infrastructure eggs into the same basket like that?