News for hackers. Stuff that matters.
Re:Lessons Learned from Twenty Years of Site Reliability Eng... by kevincox on Monday October 30, 2023 @01:49PM (#38072962)
The problem is that you can never really destroy and recreate everything. At the very least you want to preserve your user data but you probably also don't want to regularly go offline. IIRC Google used to have a dependency on starting up a new datacenter where it needed a DNS request to have some core services find each other and start up. This could be served by any other Google datacenter. But if all datacenters went offline at the same time it wouldn't be able to start. So just destroying and recreating full datacenters won't find this. You literally would have to take all of Google offline at the same time to find this by testing.

There are no comments on this comment yet.