News for hackers. Stuff that matters.

stat_throwaway

Re:Ask HN: Anyone else seeing people averaging percentiles? on Tuesday September 14, 2021 @01:28AM (#28520589)
Here's a simplified example, adapted from an actual case I saw:

We have ten servers. Each server takes 90% user requests and 10% googlebot requests. Each user request takes exactly 200 ms, each googlebot request takes exactly 500 ms.

Each server reports 80% latency of 200 ms. After averaging, it's still 200 ms, which is correct.

Now, we conjecture that by sending all googlebot requests to a single server, it can use cache more efficiently. So we change routing: now nine servers get only user requests (90% of traffic), and one server gets only googlebot requests (10% of traffic). By doing this, we speed up every request by 10 ms. Great success! Or is it?

Let's calculate average p80. 80% latency of a user server: 190 ms (everything takes exactly 190 ms here). 80% latency of a googlebot server: 490 ms. Average latency = (190 * 9 + 490 * 1) / 10 = 220 ms. Huh wait, the service became 20 ms slower! Something is wrong, maybe we should revert the rollout ...

-----

You may say "Well, now that you explained what you did, of course that's wrong." The problem is, there's no right way to average percentiles. The numbers will be always suspect. The only difference is when it fails.

As far as I can tell, the only time the numbers are right is when every server gets exactly same mix of requests and shows exactly same performance characteristic, but of course, if you already know this then you don't even need average. You can take p90 from a single server, and every other server will have the same value.

You are aggregating because you DON'T know if they are all behaving in the same way. When you're averaging percentiles, you're assuming the very thing you set out to verify.

Ask HN: Anyone else seeing people averaging percentiles?

I work at a middling SV company, and I see people taking averages of percentiles (or, even crazier, percentiles of percentiles) every day - that is, each server computes its own "50%/90%/95% latency" of all requests, sends it to a central time-series database, and the Grafana console averages them all to show a "nice" graph. They are used for everything from alerts to...

( Read More... | 11 comments | 15 points | on HN )