News for hackers. Stuff that matters.
Re:Raspberry Pi 5 by wredue on Thursday September 28, 2023 @11:22AM (#37690914)
5% is not arbitrary. It is removing massively skewing outliers who also see a massive growth outlier compared to everyone else.

Looking at wage movement as slices of where one falls gives a clearer picture.

Additionally, the reason it doesn’t make sense to remove low wage outliers while removing high wage outliers for average is because the median wage is closer to low wage outliers than it is to high wage outliers.

For example, if you take the median and then +- for your dataset (if median is 48,000, then use 0 thru 96,000), you’ll be removing more than just the top 5%, and yet, this also gives a far far far better picture of the dire economic position and what is happening with real wage movement.

Re:Raspberry Pi 5 by Gormo on Thursday September 28, 2023 @07:16PM (#37697223)
Why would you cut off a fixed percentage rather than use a standard deviation range.
Re:Raspberry Pi 5 by squeaky-clean on Thursday September 28, 2023 @07:20PM (#37697248)
It's arbitrary because why not 4% or 6%. 5 just happens to be a pleasant number but there's nothing special about it in an economic sense.
Re:Raspberry Pi 5 by Gormo on Thursday September 28, 2023 @08:12PM (#37697708)
Why would you cut off a fixed percentage rather than use a standard deviation range?
Re:Raspberry Pi 5 by ralphm on Friday September 29, 2023 @04:48AM (#37700988)
Because wages are non-Gaussian, standard deviation doesn't mean anything in this context.
Re:Raspberry Pi 5 by Gormo on Monday October 09, 2023 @10:01AM (#37820538)
Interesting. Most income graphs I've seen appear to be normally distributed -- could you explain what would cause them not to be, or provide any examples that demonstrate non-Gaussian distributions?

But if wages are non-Gaussian, how would trimming the top and bottom 5% off of your sample be any better than using standard deviation ranges to control for outliers? The assumption that outliers are to be found on the top and bottom of your range is one that seems to apply to Gaussian distributions, and doesn't necessarily hold for others, regardless of whether you are using fixed percentage values or standard deviation thresholds. For example, in a bimodal distribution, outliers might be found in the center.