How percentile approximation works and why it's more useful than averages

Re:How percentile approximation works and why it's more usef... by madars on Tuesday September 14, 2021 @12:45PM (#28527401)

Good opportunity to plug https://en.wikipedia.org/wiki/Anscombe%27s_quartet : if you don't know much about the underlying distribution, simple statistics don't describe it well.

From Wikipedia description: Anscombe's quartet comprises four data sets that have nearly identical simple descriptive statistics, yet have very different distributions and appear very different when graphed. Each dataset consists of eleven (x,y) points. They were constructed in 1973 by the statistician Francis Anscombe to demonstrate both the importance of graphing data before analyzing it, and the effect of outliers and other influential observations on statistical properties.

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by pdpi on Tuesday September 14, 2021 @01:07PM (#28527693)

There's a fun paper by Autodesk where they make datasets that look whatever way you want them to.

https://www.autodesk.com/research/publications/same-stats-di...

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by s_gourichon on Tuesday September 14, 2021 @01:19PM (#28527850)

Yes, a both funny and insightful lesson on how weak basic indicators (mean, standard deviation, correlation) can be, the interest and limits of box-plot with quartiles and whiskers, the benefit of the violin-plot.

Definitely worth a quick look.

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by djk447 on Tuesday September 14, 2021 @01:09PM (#28527704)

NB: Post author here.

This is great! So fun...will have to use in the future...

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by pdpi on Tuesday September 14, 2021 @01:29PM (#28527954)

While I have your attention...

> For the median and average to be equal, the points less than the median and greater than the median must have the same distribution (i.e., there must be the same number of points that are somewhat larger and somewhat smaller and much larger and much smaller).

[0, 2, 5, 9, 9] has both median and mean = 5, but the two sides don't really have the same distribution.

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by djk447 on Tuesday September 14, 2021 @01:35PM (#28528045)

Totally true...thoughts on how I could rephrase? I guess it's more the "weight" of points greater than and less than the median should be the same, so symmetric distributions definitely have it, asymmetric may or may not. Definitely open to revising...

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by pdpi on Tuesday September 14, 2021 @01:41PM (#28528116)

> symmetric distributions definitely have it, asymmetric may or may not

Doesn't have to be any more complicated than that. It's more a curio than an important point anyway :)

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by doctorsher on Tuesday September 14, 2021 @01:05PM (#28527671)

This is excellent information, thank you for posting this! I was not familiar with this example previously, but it is a perfect example of summary statistics not capturing certain distributions well. It's very approachable, even if you had to limit the discussion to mean and variance alone. Bookmarked, and much appreciated.

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by djk447 on Tuesday September 14, 2021 @12:50PM (#28527464)

NB: Post author here.

That's really nifty, wish I'd heard about it earlier. Might go back and add a link to it in the post at some point too! Very useful. Definitely know I wasn't breaking new ground or anything, but fun to see it represented so succinctly.

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by hoseja on Wednesday September 15, 2021 @02:18AM (#28535828)

>importance of graphing data before analyzing it

Very discouraging if one is trying to analyze data algorithmically. Often when faced with a problem in statistics, the answer is: "Look at the graph and use intuition!".

[ Parent | on HN ]

Re:How percentile approximation works and why it's more usef... by potatoman22 on Tuesday September 14, 2021 @10:39PM (#28534571)

Interesting, thanks for the concept. What's one to do for high dimensional data then?

[ Parent | on HN ]