Re:Finding a problem at the bottom of the Google stack
by Nextgrid on Sunday March 15, 2020 @12:12PM
(#22584209)
My suggestion is that the on-site people should have alerts about hardware events. Unless it's a failing sensor, there shouldn't be any false positives - if a machine is exceeding its recommended operating temperature it's still not good and warrants an investigation even if it isn't yet causing problems (in fact the point of these alerts is to catch and fix these issues before they cause any user-facing issues).
[ Parent
| on HN ]
Re:Finding a problem at the bottom of the Google stack
by Forge36 on Sunday March 15, 2020 @01:44PM
(#22585040)
That I like. From the photos it's also obvious the rack is tipping.
I'm curious why someone isn't doing a regular walkthrough.
There could also be sounds of some hardware failures.
A remote SRE shouldn't need to monitor the hardware health, an on-site person should have caught this sooner.
I wonder how the person responsible for fixing/replacing the wheels felt about the follow-up
Re:Finding a problem at the bottom of the Google stack
by cameronbrown on Sunday March 15, 2020 @02:17PM
(#22585379)
Sure - the issue is we (the public reading this) don't know when this actually physically failed. It could be that SRE picked up on this before a scheduled walkthrough (of which I'm sure occurs) happened.
Disclaimer: Google employee. No knowledge of this event beyond the blog post.