Re:Feb 27 2023 GCP Outage Incident Report
by fulafel on Tuesday March 07, 2023 @11:58AM
(#35057693)
I wonder how much people in the know really believe in singling out a single root cause to these HA system failures.
[ Parent
| on HN ]
It’s really odd to see comments like this faded out from downvotes. Anyone from the devops or SRE or distributed systems would ask the same.
For example why are there no processes to check for snapshot integrity or if they are, why were they not used?
Re:Feb 27 2023 GCP Outage Incident Report
by pgwhalen on Tuesday March 07, 2023 @01:11PM
(#35058835)
It’s downvoted because it’s a straw man, the linked article isn’t suggesting there was a single root cause.
Re:Feb 27 2023 GCP Outage Incident Report
by verdverm on Tuesday March 07, 2023 @12:55PM
(#35058593)
Did you read it?
They pointed out several issues that caused this and several mitigations to prevent it from happening again.
I have yet to see anyone else with public RCAs as good as Google Cloud
Yes - under the root cause heading isn't the following raised as "the" root cause?
> During a routine update to the critical elements snapshot data, an incomplete snapshot was inadvertently shared which removed several sites from the topology map.
Re:Feb 27 2023 GCP Outage Incident Report
by verdverm on Tuesday March 07, 2023 @01:25PM
(#35059043)
Yes, that is one sentence in the full analysis which describes the core of what happened. There are other sentences which describe several contributing factors.
Re:Feb 27 2023 GCP Outage Incident Report
by sillybov3456 on Tuesday March 07, 2023 @02:01PM
(#35059599)
Right, the thing about root causes is that you can always keep digging. For instance why was an incomplete snapshot shared? And then the why for that why, and on and on until you reach the singularity at the beginning of the universe, which can logically be the only real root cause of anything. Root cause just means whatever is enough to make your boss satisfied.
Re:Feb 27 2023 GCP Outage Incident Report
by notimetorelax on Tuesday March 07, 2023 @01:54PM
(#35059497)
Look at the ‘Remediation and Prevention’ section for the fixes. Root cause of an incident is always one, but means to prevent it are multiple.
Right, but is it the right mental model to demand a root cause to be delivered as the outcome to the investigation, when there are lots of things going wrong and process & architecture problems that make the involved failures fatal.
Re:Feb 27 2023 GCP Outage Incident Report
by notimetorelax on Tuesday March 07, 2023 @04:23PM
(#35061453)
This follows Google’s published post mortem template: https://sre.google/sre-book/example-postmortem
What you’re saying can go into the lessons learned section.
Re:Feb 27 2023 GCP Outage Incident Report
by fulafel on Thursday March 09, 2023 @06:22AM
(#35080227)
That template is a bit better wrt the cognitive problem in question since the heading is "root causes" (plural). So it's more compatible with the epistemology that there are lots of interacting evens and properties and doesn't frame your thinking around pinpointing a single event in the complex graph of interacting processes leading to a bad system state. (it's more naturally a graph than a tree whcih is implied by the classic RCA model)
Well, I don’t think there is any question about it. It can only be attributable to human error. This sort of thing has cropped up before, and it has always been
due to human error.
Done.
Re:Feb 27 2023 GCP Outage Incident Report
by FigmentEngine on Tuesday March 07, 2023 @03:25PM
(#35060684)
tldr; no system failure is human error. if a human can cause this, then your system lacks adequate controls and mechanism. the root cause is the lack of controls, not the human error
Building a system without sufficient controls is a classic (human) error in system design.