News for hackers. Stuff that matters.
Re:Feb 27 2023 GCP Outage Incident Report by fulafel on Tuesday March 07, 2023 @11:58AM (#35057693)
I wonder how much people in the know really believe in singling out a single root cause to these HA system failures.
Re:Feb 27 2023 GCP Outage Incident Report by nailer on Tuesday March 07, 2023 @12:32PM (#35058257)
It’s really odd to see comments like this faded out from downvotes. Anyone from the devops or SRE or distributed systems would ask the same.

For example why are there no processes to check for snapshot integrity or if they are, why were they not used?

Re:Feb 27 2023 GCP Outage Incident Report by pgwhalen on Tuesday March 07, 2023 @01:11PM (#35058835)
It’s downvoted because it’s a straw man, the linked article isn’t suggesting there was a single root cause.
Re:Feb 27 2023 GCP Outage Incident Report by verdverm on Tuesday March 07, 2023 @12:55PM (#35058593)
Did you read it?

They pointed out several issues that caused this and several mitigations to prevent it from happening again.

I have yet to see anyone else with public RCAs as good as Google Cloud

Re:Feb 27 2023 GCP Outage Incident Report by fulafel on Tuesday March 07, 2023 @01:12PM (#35058839)
Yes - under the root cause heading isn't the following raised as "the" root cause?

> During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.

Re:Feb 27 2023 GCP Outage Incident Report by verdverm on Tuesday March 07, 2023 @01:25PM (#35059043)
Yes, that is one sentence in the full analysis which describes the core of what happened. There are other sentences which describe several contributing factors.
Re:Feb 27 2023 GCP Outage Incident Report by sillybov3456 on Tuesday March 07, 2023 @02:01PM (#35059599)
Right, the thing about root causes is that you can always keep digging. For instance why was an incomplete snapshot shared? And then the why for that why, and on and on until you reach the singularity at the beginning of the universe, which can logically be the only real root cause of anything. Root cause just means whatever is enough to make your boss satisfied.
Re:Feb 27 2023 GCP Outage Incident Report by notimetorelax on Tuesday March 07, 2023 @01:54PM (#35059497)
Look at the ‘Remediation and Prevention’ section for the fixes. Root cause of an incident is always one, but means to prevent it are multiple.
Re:Feb 27 2023 GCP Outage Incident Report by fulafel on Tuesday March 07, 2023 @02:31PM (#35060022)
Right, but is it the right mental model to demand a root cause to be delivered as the outcome to the investigation, when there are lots of things going wrong and process & architecture problems that make the involved failures fatal.
Re:Feb 27 2023 GCP Outage Incident Report by notimetorelax on Tuesday March 07, 2023 @04:23PM (#35061453)
This follows Google’s published post mortem template: https://sre.google/sre-book/example-postmortem

What you’re saying can go into the lessons learned section.

Re:Feb 27 2023 GCP Outage Incident Report by fulafel on Thursday March 09, 2023 @06:22AM (#35080227)
That template is a bit better wrt the cognitive problem in question since the heading is "root causes" (plural). So it's more compatible with the epistemology that there are lots of interacting evens and properties and doesn't frame your thinking around pinpointing a single event in the complex graph of interacting processes leading to a bad system state. (it's more naturally a graph than a tree whcih is implied by the classic RCA model)
Re:Feb 27 2023 GCP Outage Incident Report by toast0 on Tuesday March 07, 2023 @02:12PM (#35059763)
Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been due to human error.

Done.

Re:Feb 27 2023 GCP Outage Incident Report by FigmentEngine on Tuesday March 07, 2023 @03:25PM (#35060684)
tldr; no system failure is human error. if a human can cause this, then your system lacks adequate controls and mechanism. the root cause is the lack of controls, not the human error
Re:Feb 27 2023 GCP Outage Incident Report by toast0 on Tuesday March 07, 2023 @03:44PM (#35060947)
Building a system without sufficient controls is a classic (human) error in system design.
Re:Feb 27 2023 GCP Outage Incident Report by fulafel on Thursday March 09, 2023 @10:52AM (#35082530)
I chuckled in agreement but then I started thinking if it could be actually a better term. Human error analysis might be a little blamey.. but availability design bug or something might be good.