News for hackers. Stuff that matters.
Re:Finding a problem at the bottom of the Google stack by joshuamorton on Sunday March 15, 2020 @06:31PM (#22587421)
Lots of things that should "never" happen start to happen with frightening regularity once you have enough machines.

Few of them are worth paging someone.

Re:Finding a problem at the bottom of the Google stack by dijit on Sunday March 15, 2020 @06:40PM (#22587488)
Those are two extremes.

For me, when a machine reports as abnormal it will decommission itself and file a ticket to be looked at, we have enough extra capacity that it’s ok to do this for 5% of compute before we have issues, and we have an alert if it gets close to that.

If you’re doing anything other than that then either your machines are abnormal and functioning, which is scary because they’re now in an unknown state- or you’re able to just throw out anything that seems weird, which might also hide issues with the environment.

Re:Finding a problem at the bottom of the Google stack by Dylan16807 on Monday March 16, 2020 @07:14AM (#22591700)
Whether you page someone or not, a machine hitting thermal throttling repeatedly should be just as notable as the abnormal failure rate that started the investigation.