Re:Google Bug Bounty – The $5k Error Page
by londons_explore on Friday May 19, 2017 @11:39AM
(#14376724)
Boss: Median latency is 100ms and 99.9th percentile latency is 1 second.
Nobody ever asks about that 0.1%...
Re:Google Bug Bounty – The $5k Error Page
by developer2 on Saturday May 20, 2017 @12:19PM
(#14383072)
When that 0.1% - or even 0.001% - are 5-60 second requests, you have a bomb waiting to go off. There really is a massive difference when you are operating at the scale of Google. If the median is 100ms, the maximum acceptable time - 100th percentile - is likely below 200ms. A three nines percentile that is 10x the median isn't a good thing at large scale. Perfect consistency is more important than statistics. A small scale service deployed on my-little-unused-tool.com that receives a few requests/minute is an entirely different ballgame.
When you operate at the scale of Google, everything is expected to be airtight; outliers should not be possible. It wouldn't surprise me if their monitoring systems are built without the ability to "massage" (ie: manipulate) statistics, as it is a terrible practice. I don't think a statistician who relies on ignoring outliers would last long working for Google. They're not doing their job if the only thing they care about is silencing warnings to make pretty graphs that falsely show everything is running smoothly. Their job is to work with the truth - not manufacture little white lies to appease management.