News for hackers. Stuff that matters.
Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by imtringued on Thursday March 19, 2020 @09:31AM (#22627270)
Every time I hear about percentiles I am thinking: Why not just show the whole distribution instead of picking a few values? I was immediately thinking of just showing the latency distribution in a histogram and was pleased by the article doing exactly that. Of course graphing percentiles over time is much easier because they just represent a single value. Percentiles are very useful for finding latency spikes but not that good for analyzing them.
Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by TeMPOraL on Thursday March 19, 2020 @09:44AM (#22627402)
> Of course graphing percentiles over time is much easier because they just represent a single value.

Ridgeline plots (joyplots) are severely underutilized.

Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by mncharity on Thursday March 19, 2020 @10:16AM (#22627673)
Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by hinkley on Thursday March 19, 2020 @09:35PM (#22633687)
There are also violin plots, which are like a cross between a box plot and a ridgeline plot.

https://en.wikipedia.org/wiki/Violin_plot

Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by kasey_junk on Thursday March 19, 2020 @10:01AM (#22627552)
I generally love joyplots but haven’t seen a great use of them showing latency distribution over time.

Do you know of any examples?

Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by kqr on Thursday March 19, 2020 @11:22AM (#22628311)
Heatmaps sorta-kinda solve the problem of plotting histograms over time, except of course it's harder to detect differences in brightness than size.
Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by GGfpc on Thursday March 19, 2020 @12:34PM (#22629114)
I tend to use cumulative distribution functions when comparing distributions.

Histograms also tend to be quite easy to manipulate

Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by OliverJones on Thursday March 19, 2020 @11:12AM (#22628226)
Yes, absolutely, show the distribution whenever you can.

But consider production alerting purposes. For example, consider database user-lookup time. Use the 95th percentile, or even the maximum value to compare with an alert threshold. In many (most?) real-world cases system trouble shows up as a few lookup-time outliers. If you only look at averages, or even medians, you miss those outliers.

Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by pdpi on Thursday March 19, 2020 @11:06AM (#22628161)
Histograms, too, bin values and thus show only a few values instead of the whole distribution.

In fact, the only big difference between histograms and percentiles are that they're sort of the same thing on a different axis.

Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by djspoons on Thursday March 19, 2020 @11:20AM (#22628293)
Well, yeah, there's not much difference between a histogram and hundreds-of-percentiles. But there's a huge difference between a histogram and just a single percentile (whether it's p50, p95, p99, or whatever).

If you just want to know _if_ something changed then maybe one percentile is ok (though actually: not great). But if you are trying to figure out _what_ happened (or is happening!), a histogram is really important. Most of the time there are discrete behaviors/factors that are driving performance: timeouts, cache hits/misses, one overloaded host, a canary. The shape of the histogram will help you see those in a way that a single percentile can't.

Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by djspoons on Thursday March 19, 2020 @11:45AM (#22628536)
Actually one of my coworkers gave a talk at Facebook Performance summit on this: https://www.youtube.com/watch?v=EG7Zhd6gLiw (Disclaimer: brief product pitch in the first minute as part of the speaker intro)
Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by lonelappde on Thursday March 19, 2020 @09:43AM (#22627393)
The main objection is that it's hard to get a 3D graph of the distribution over time. But it's still worth drawing some snapshots of the distribution.
Re:Lies, Damned Lies, and Averages: Perc50, Perc95 Explained... by meatmanek on Thursday March 19, 2020 @08:12PM (#22633174)
Heat maps (showing histogram bin count as color/intensity instead of bar height) and scatterplots (potentially of a subset of data points) are both relatively straightforward ways to visualize distributions over time.