Finding a problem at the bottom of the Google stack

Re:Finding a problem at the bottom of the Google stack by popinman322 on Sunday March 15, 2020 @05:28PM (#22586928)

It's Google; they have sufficient ML capacity to filter cases. It might require additional manpower in the short term, but it also lowers the likelihood of infrastructure failure long-term.

What might not make sense, resource-wise, to smaller companies does start to make sense at their scale.

[ Parent | on HN ]

Re:Finding a problem at the bottom of the Google stack by jldugger on Monday March 16, 2020 @01:21AM (#22590237)

Anomaly detection on timeseries data is surprisingly bad. The data tends to be bad in all the wrong ways -- seasonal, and not normally distributed.

And worse, you have no training data, so ML model you train simply learns whatever badness is allowed to persist in production as normal.

[ Parent | on HN ]

Re:Finding a problem at the bottom of the Google stack by cranekam on Monday March 16, 2020 @03:06AM (#22590663)

I worked at a FAANG and every now and then someone proposed or tried to make an anomaly detection system. They never worked well. It’s an extremely difficult thing to get right. Better to just have good monitoring of whether your system is responsive/available/reliable and ways to quickly work out why it’s not.

[ Parent | on HN ]