News for hackers. Stuff that matters.
Re:Outlier Detection at Netflix by chrisfosterelli on Tuesday July 14, 2015 @05:16PM (#9888117)
Very cool work! You mentioned that in a lot of cases, basic thresholds didn't work very well. I understand that setting a direct cutoff would trigger many false positives or not enough true positives, but what about time-based thresholds?

I had always figured timed filters like "above X error rate for Y occurrences in Z time" may work. Threshold filters can be much more complex, I'm curious if you tried different types of threshold filters and if they worked or not.

Re:Outlier Detection at Netflix by diab0lic on Tuesday July 14, 2015 @05:23PM (#9888154)
One of the more complex situations that this solution attempts to address is the situation where the entire cluster surges upwards.

Imagine that service A depends on service B, and service B begins experiencing an issue. The outlier detection system that service A has setup will see the entire cluster surge upwards on errors. This would trip a traditional threshold based approach, but the last thing we'd want to do is terminate servers in service A. In this case the entire cluster moving upwards sets the context that there is a larger scale problem going on (nothing is outlying), whereas if only a few move upwards (become outliers) we have a localized issue we can fix.

Meanwhile service B through either outlier or threshold alerts is paged and dealing with their issue.

As a side note we do something similar to time sensitive thresholds when doing threshold based alerting on differences between the observed data and a double exponential smoothing fit. Parameter selection is an issue at times in this case, we talk about this a bit in an older blog post on Stream Starts per Second (SPS): http://techblog.netflix.com/2015/02/sps-pulse-of-netflix-str...

Hope that adds a little context for you. :)

Re:Outlier Detection at Netflix by chrisfosterelli on Tuesday July 14, 2015 @05:27PM (#9888174)
That makes a lot of sense, thanks for the answer!
Re:Outlier Detection at Netflix by diab0lic on Tuesday July 14, 2015 @05:44PM (#9888274)
No problem, thanks for taking the time to read it! :D