Re:Google Cloud Is Down
by aaronblohowiak on Sunday June 02, 2019 @10:16PM
(#20079903)
We went multi-region as a result of the 2012 inc. source: I now manage the team responsible for performing regional evacuations (shifting traffic and scaling the savior regions).
That sounds fascinating! How often does your team have to leap into action?
Re:Google Cloud Is Down
by aaronblohowiak on Sunday June 02, 2019 @11:25PM
(#20080172)
We don’t usually discuss the frequency of unplanned failovers, but I will tell you that we do a planned failover at least every two weeks. The team also uses traffic shaping to perform whole system load tests with production traffic, which happens quarterly.
Re:Google Cloud Is Down
by justinator on Monday June 03, 2019 @01:11AM
(#20080557)
Do you do any chaos testing? Seems like it would slot right in, there.
I'd say yes. I heard about this tool just a week ago at a developer conference.
Netflix was a pioneer of chaos testing, right? https://en.m.wikipedia.org/wiki/Chaos_engineering
Re:Google Cloud Is Down
by aaronblohowiak on Monday June 03, 2019 @05:32PM
(#20089053)
Re:Google Cloud Is Down
by arainwater on Monday June 03, 2019 @03:20AM
(#20081096)
they have invented the term, so probably yes :)
I think some Google engineers published a free Meap book on service relatability and uptime guarantees. Seemingly counterintuitive, scheduling downtime, without other teams’ prior knowledge, encourages teams to handle outages properly and reduce single points of failure, among other things.
Service Reliability Engineering is on OReilly press. It's a good book. Up there with ZeroMQ and Data Intensive Applications as maybe the best three books from OReilly in the past ten years.
Derp, Site Reliability Engineering.
"After a 2012 storm-related power outage at Amazon during which Netflix suffered through three hours of downtime, a Netflix engineer noted that the company had begun to work with Amazon to eliminate “single points of failure that cause region-wide outages.” They understood it was the company’s responsibility to ensure Netflix was available to entertain their customers no matter what. It would not suffice to blame their cloud provider when someone could not relax and watch a movie at the end of a long day."
https://www.networkworld.com/article/3178076/why-netflix-did...