News for hackers. Stuff that matters.
Re:Finding a problem at the bottom of the Google stack by dlandis on Sunday March 15, 2020 @02:38PM (#22585560)
> The SRE tracked the problem all the way from an external, front-end system down to the hardware that holds up the machines.

I agree, this does seem like an incredibly circuitous and inefficient route to diagnosing the issue, as opposed to the hardware team simply receiving an alert that a server was continuously in an overheated, and then fixing it.

Re:Finding a problem at the bottom of the Google stack by jldugger on Sunday March 15, 2020 @03:04PM (#22585771)
In my experience the breakfix team has no shortage of work to be done relative to staffing. But they have no insight into the relative priorities of things. That's where your customer metrics and in depth investigation helps.

It's entirely possible a server overheating ticket was in the queue so to speak, and the investigation ends up re-prioritizing an alarm that recovered before anyone was on site to fix it.