8040
Re:Lessons Learned from Twenty Years of Site Reliability Engineering
on Friday October 27, 2023 @08:46PM (#38045829)
I would like to take this moment to really highlight "Recovery mechanisms should be fully tested before an emergency". As the unexpected SRE at Google who became known by entire company for using a double negative incorrectly, it is something very important to do right away.
For those Googlers curious, you can search my username internally for how I generated more impact then could be measured.
Re:Monarch: Google’s Planet-Scale In-Memory Time Series Database
on Monday May 23, 2022 @12:05AM (#31475245)
https://status.cloud.google.com/incident/google-stackdriver/... was one of the public incident reports around it.
Re:Monarch: Google’s Planet-Scale In-Memory Time Series Database
on Monday May 23, 2022 @12:02AM (#31475229)
while technically correct, it was degraded in a number of interesting ways that one might describe as broken, but still within SLO
Re:Monarch: Google’s Planet-Scale In-Memory Time Series Database
on Saturday May 14, 2022 @02:05PM (#31380620)
I broke this once several years ago. I even use the incident number in my random usernames to see if a Googler recognizes it.