Lessons Learned from Twenty Years of Site Reliability Engineering

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by throwawaaarrgh on Friday October 27, 2023 @04:28PM (#38043477)

The solution to this has to be done earlier, but it's simple: start a habit of destroying and recreating everything. If you wait to start doing this, it's very painful. If you start doing it at the very beginning, you quickly get used to it, and breaking changes and weird dependencies are caught early.

You can even do this with hardware. It changes how you architect things, to deal with shit getting unplugged or reset. You end up requiring more automation, version control and change management, which speeds up and simplifies overall work, in addition to preventing and quickly fixing outages. It's a big culture shift.

[ Parent | on HN ]

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by ahmedfromtunis on Friday October 27, 2023 @08:17PM (#38045617)

My SaaS app is small even compared to some desktop apps. But at least once a year, I try to reboot it from scratch.

Of course this is way easier for a nano-scale app, but I love the feeling of knowing that it can be started from any server in the world in less than 10 minutes (including copying the data).

I also make sure there are 0 errors even with a clean slate database.

For some reason I can't understand, this gives me some joy.

[ Parent | on HN ]

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by jeremyjh on Saturday October 28, 2023 @10:05AM (#38049898)

That really doesn't address GP's concerns at all, which are only concerns for the hyperscale cloud providers. Because they use their global infrastructure to turn on a data center that has been down, we don't know if they have a way to restart the entire planet. It is impossible to test this, because that would cause a global outage that could possibly be permanent.

[ Parent | on HN ]

Re:Lessons Learned from Twenty Years of Site Reliability Eng... by kevincox on Monday October 30, 2023 @01:49PM (#38072962)

The problem is that you can never really destroy and recreate everything. At the very least you want to preserve your user data but you probably also don't want to regularly go offline. IIRC Google used to have a dependency on starting up a new datacenter where it needed a DNS request to have some core services find each other and start up. This could be served by any other Google datacenter. But if all datacenters went offline at the same time it wouldn't be able to start. So just destroying and recreating full datacenters won't find this. You literally would have to take all of Google offline at the same time to find this by testing.

[ Parent | on HN ]