News for hackers. Stuff that matters.
Re:How the US Air Force Ditched the Average and Saved Lives by cm2187 on Saturday July 02, 2022 @05:35AM (#31957532)
The other massive misuse of average I keep hearing is people referring to the life expectancy at the XVIII century. Yeah it was 40 something but not because it was very rare for people to reach 60, but because the average is dragged down by the high infant mortality rate. Once you reached 20, your life expectancy was a bit lower than today, but it was common for people to reach 60.

And in general people are really bad at thinking in term of distributions. If you discuss averages or percentiles, people identify to that metric like if it applied to every individual of that distribution. That makes the debate on D&I particularly unproductive.

Re:How the US Air Force Ditched the Average and Saved Lives by inglor_cz on Saturday July 02, 2022 @06:50AM (#31957894)
People don't generally understand premodern demography. We have fairly good demographic record of England, where detailed birth and death records were kept.

As of the 18th century, about 30 per cent of people died before reaching adulthood, but this percentage seems to be much higher in urban areas, which acted like population sinks. (Pathogens were really concentrated there.) Once you lived to be 20, you had more than even chance to live to 40, and a good (AFAIK over 35 per cent) chance to live to 60, but the drop-off after that was steep, 70 y.o.s were already uncommon and 80 y.o.s very rare.

Interestingly, cardinals and popes lived significantly longer, 70-somethings were a common sight in conclaves. Easier life, no military threat, good water, almost no risk of famines.

I am writing this from the top of my head, so precise values may differ from my handwaving. But I believe that the values are roughly correct.

Re:How the US Air Force Ditched the Average and Saved Lives by elephant_burger on Saturday July 02, 2022 @07:07AM (#31957999)
Looking at deaths globally by age as a distribution shows how the average can mislead https://www.google.com/search?q=deaths+globally+by+age&rlz=1...
Re:How the US Air Force Ditched the Average and Saved Lives by atwood22 on Saturday July 02, 2022 @11:38AM (#31959931)
Hmm based on your comment, I think people understand better than you think. Obviously if the life expectancy is 40, then some people live longer. Half of all people who live to 20 dying before 40 is probably what most people think of when they think about a 40 year life expectancy.
Re:How the US Air Force Ditched the Average and Saved Lives by Darmody on Saturday July 02, 2022 @01:07PM (#31960685)
From my experience, people think most people back then lived around 40 years, while some lived less and some lived more.
Re:How the US Air Force Ditched the Average and Saved Lives by Amezarak on Saturday July 02, 2022 @12:53PM (#31960600)
I don’t know specifically about that time period, but there have been dips and peaks in life expectancy over time. In the latter half of the 19th century in England, life expectancy at adulthood was the same or better than it is today.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2672390/

There was actually a major, severe DECLINE after that associated with globalization, trade, and further industrialization, which is where many charts begin.

Think about how incredible that is - nothing we would recognize as modern medicine, but adults living just as long. I can’t think of any greater indictment of our health in modern civilization, especially given the amount of resources we expend. I mean, just think about - aside from the improvements in child mortality which are almost entirely just from now nearly century old antibiotics and vaccines technology, we spend 20% of our GDP to achieve the same results as the 1850s English who got them for next to nothing in comparison. Progress!

Re:How the US Air Force Ditched the Average and Saved Lives by bergenty on Sunday July 03, 2022 @03:23AM (#31965809)
If you want to look at the silver lining, we have much easier lives and access to a tremendous number of cheap, rich foods (albeit unhealthy) and plenty of cheap recreation and drugs. We have worse living habits than them and spend much more time in sedentary positions. It’s probably impressive we can keep the same life expectancy with all those factors.
Re:How the US Air Force Ditched the Average and Saved Lives by hawaiianbrah on Sunday July 03, 2022 @04:23PM (#31970688)
> we spend 20% of our GDP to achieve the same results as the 1850s English who got them for next to nothing in comparison.

I have to assume that our lives are overall much easier medically, though. Breaking a limb (or even losing one) is not nearly as big of a deal now than it probably was then, for instance.

Re:How the US Air Force Ditched the Average and Saved Lives by pxmpxm on Saturday July 02, 2022 @12:50PM (#31960562)
IIRC bulk of the gain in average life expectancy over the last 2 centuries came from reducing infant mortality, which had outside impact on the aggregate mean.
Re:How the US Air Force Ditched the Average and Saved Lives by kqr on Saturday July 02, 2022 @07:25AM (#31958102)
Our World In Data has a plot of conditional life expectancy throughout recent history. It's one of my favourite plots of all time not because it's important, but informative and intuitive.

https://ourworldindata.org/uploads/2013/05/Life-expectancy-b...

Re:How the US Air Force Ditched the Average and Saved Lives by cm2187 on Saturday July 02, 2022 @07:36AM (#31958174)
That's a very interesting plot thank you.

Another very powerful demographic chart that I lost (in case someone knows where it is) is a 3 dimensional chart. Y axis is children per women, X axis is infant mortality, and then there is a slider to look at the evolution through time. All countries are plotted as points.

The reason it is powerful is because by moving the slider you can see the evolution through time. For most now developed countries, infant mortality and children per women both reduced simultaneously as the progress of medicine and the evolution of society have been very gradual. But african countries are following a different path, where the infant mortality collapsed but is not followed by a reduction of children per woman (only a few countries have started to take that turn). That is basically all you need to know to explain the major demographic shift we are about to observe in Africa.

Re:How the US Air Force Ditched the Average and Saved Lives by kqr on Saturday July 02, 2022 @07:54AM (#31958279)
Re:How the US Air Force Ditched the Average and Saved Lives by cm2187 on Saturday July 02, 2022 @07:58AM (#31958313)
It's exactly that, thank you. The version I saw didn't change the scale when you move the slider which makes it easier to see what is going on, but I am pretty sure it must be earlier version of that chart.
Re:How the US Air Force Ditched the Average and Saved Lives by WanderPanda on Saturday July 02, 2022 @09:28AM (#31958810)
Nice plot, we should work on steepening those curves to a slope of greater than one
Re:How the US Air Force Ditched the Average and Saved Lives by xeromal on Saturday July 02, 2022 @03:04PM (#31961452)
Is that drop in the earlier 1900s localized to the young the spanish flu?
Re:How the US Air Force Ditched the Average and Saved Lives by late2part on Sunday July 03, 2022 @01:47PM (#31969522)
I was thinking WW1
Re:How the US Air Force Ditched the Average and Saved Lives by wardedVibe on Saturday July 02, 2022 @11:14AM (#31959720)
The Spanish fly/great war dip is staggering
Re:How the US Air Force Ditched the Average and Saved Lives by smcl on Saturday July 02, 2022 @01:58PM (#31961022)
And the sharp increase around the mid 1940s is quite remarkable too. I was wondering what caused that, I guess the polio vaccination started getting rolled out but I don't if it was that big of a cause of death.
Re:How the US Air Force Ditched the Average and Saved Lives by kqr on Saturday July 02, 2022 @02:36PM (#31961275)
"Methods for mass production of penicillin were patented by Andrew Jackson Moyer in 1945."

Antibiotics are a lifesaver to reckon with.

Re:How the US Air Force Ditched the Average and Saved Lives by smcl on Saturday July 02, 2022 @03:17PM (#31961533)
Ahhh of course :D
Re:How the US Air Force Ditched the Average and Saved Lives by Phiwise_ on Saturday July 02, 2022 @09:50AM (#31958959)
People don't generally understand how statistical metrics are supposed to work. The whole point of statistics is to give you reasonable expectations for the possibilities of a space that is both full of variation but well-explored based on what particulars you know already. The problem is that most people don't realize the arithmetic mean is intended to give you the expectations associated with not knowing essentially anything. It's the map you give a stranger in a strange land (am I using that reference right? I haven't actually read the book yet); the way I explain how to get somewhere in my hometown to a family member isn't just not the same as the way I would to somewhere near my new job, but runs a worryingly high chance of making things less clear than if they just tried to do it themselves (well, at least, before the days of google maps. Why does technology have to be so good at ruining metaphors?). When you have, or will have, information, working without it is just asking to get tripped up. This the brilliance of the adjustable cockpits example: The Air Force leadership assumed they couldn't set cockpits to fit their pilots because they didn't currently know which pilot would fly which plane, forgetting that no plane takes off in the Air Force without the leadership telling a particular pilot to get in a particular plane. Once they adopted the life-like "What do I know about Y given what I (will) know about X" approach over the card-table-like "What do I know about Y given I can't know anything about X", seemingly untouchable problems immediately resolved out almost on their own.

We see this immediately and intuitively from the great conditional graph in this comment: https://news.ycombinator.com/item?id=31958102

Re:How the US Air Force Ditched the Average and Saved Lives by nostromo95 on Saturday July 02, 2022 @12:53PM (#31960596)
>am I using that reference right?

Looks right to me! Although interestingly it’s unclear which book / Book you’re referring to, as the expression comes from Exodus 2:22:

“And she bare him a son, and he called his name Gershom: for he said, I have been a stranger in a strange land.”

Re:How the US Air Force Ditched the Average and Saved Lives by hawaiianbrah on Sunday July 03, 2022 @04:26PM (#31970705)
Maybe that’s the book they were referring to!
Re:How the US Air Force Ditched the Average and Saved Lives by melling on Saturday July 02, 2022 @07:08AM (#31958009)
That’s because all probabilities are conditional, and people don’t consider that.

These two things helped me to understand probability better.

The Signal and the Noise:

https://www.amazon.com/Signal-Noise-Many-Predictions-Fail-bu...

The videos for Harvard Statistics 110:

https://projects.iq.harvard.edu/stat110/home

Re:How the US Air Force Ditched the Average and Saved Lives by lawrenceyan on Saturday July 02, 2022 @12:44PM (#31960505)
This is a great quote. I might have to steal this.
Re:How the US Air Force Ditched the Average and Saved Lives by kurupt213 on Saturday July 02, 2022 @11:55AM (#31960076)
Blood pressure medicine is probably one of the biggest factors in pushing adult life expectancy past 60. Antibiotics get all the credit but I think their biggest demographic impact was reducing child mortality from middle ear infections.
Re:How the US Air Force Ditched the Average and Saved Lives by naveen99 on Sunday July 03, 2022 @10:46AM (#31968077)
Very skeptical of this claim.
Re:How the US Air Force Ditched the Average and Saved Lives by RcouF1uZ4gsC on Saturday July 02, 2022 @09:32AM (#31958838)
> Once you reached 20, your life expectancy was a bit lower than today, but it was common for people to reach 60.

If you were a man. If you were a woman, there was still the high risk of maternal mortality. Add to that the large number of births, and a significant amount of women died during pregnancy/birth/post-birth.

Re:How the US Air Force Ditched the Average and Saved Lives by elephant_burger on Saturday July 02, 2022 @06:41AM (#31957836)
To your point, the global “average” life expectancy since the 1700s to today visualized https://ourworldindata.org/grapher/life-expectancy?tab=chart...
Re:How the US Air Force Ditched the Average and Saved Lives by bigDinosaur on Saturday July 02, 2022 @06:45AM (#31957860)
How to mislead with data 101!
Re:How the US Air Force Ditched the Average and Saved Lives by jwlit on Saturday July 02, 2022 @07:07AM (#31958000)
Indeed, though see the article linked at the bottom of that one, which goes into more of the nuance and has distribution curves etc: https://ourworldindata.org/life-expectancy-how-is-it-calcula...
Re:How the US Air Force Ditched the Average and Saved Lives by ackfoobar on Saturday July 02, 2022 @02:19PM (#31961138)
> If you discuss averages or percentiles, people identify to that metric like if it applied to every individual of that distribution. That makes the debate on D&I particularly unproductive.

Damore included a graph of overlapping bell curves in his document to illustrate this point. Yet quite a lot of critiques I saw don't seem to understand that.

Re:How the US Air Force Ditched the Average and Saved Lives by jameshart on Saturday July 02, 2022 @03:07PM (#31961470)
Overlapping bell curves are ripe for misinterpretation.

Look at the male/female height graphs on https://www.usablestats.com/lessons/normal for example (among the first examples that google came up with).

The thing that stands out is the difference between the bell curves - the area under the male height curve that is not under the female one. (Indeed, on this page, the way they drew their histogram version of the curve they explicitly drew attention to this area)

But this area isn’t representative of a meaningful population.

It’s just the sum of the excess number of men of a given height over and above the number of women of the same height.

Crucially, the vast majority of men accounted for within that population are still shorter than some women.

Unless you are, according to these numbers, over 77” tall - ie, over 6’5” - then there exist women who are taller than you.

Admittedly the population of people taller than you certainly skews heavily male - but for any randomly selected group of men, in most cases it is possible to find a group of just as many women who are all taller.

I personally find that the overlapping bell curve illustration obscures that understanding, making it emphasize more that a small number of below-average men are still taller than some women, and completely hiding the tail of outlier men on the left who are shorter than the vast majority of women…

Stacked histograms are a better way to visualize this kind of faceted distribution - but even that has issues.

Re:How the US Air Force Ditched the Average and Saved Lives by ackfoobar on Saturday July 02, 2022 @03:22PM (#31961567)
> the area under the male height curve that is not under the female one.

It's not obvious to me. Yes the height of the male graph is lower, but is spread wider.

> It’s just the sum of the excess number of men of a given height over and above the number of women of the same height.

It's simply that the female histogram covers part of the male histogram. It's a bit confusing. If I make the graph I will colour the overlapped parts a different colour.

This is not a problem when we show only the bell curves.

Re:How the US Air Force Ditched the Average and Saved Lives by jarenmf on Saturday July 02, 2022 @06:50AM (#31957892)
Indeed, I keep hearing this countless times. People genuinely think that people died in their 30s two centuries ago.
Re:How the US Air Force Ditched the Average and Saved Lives by olddustytrail on Saturday July 02, 2022 @01:23PM (#31960786)
They did. Large numbers of them. Often in childbirth.
Re:How the US Air Force Ditched the Average and Saved Lives by wizofaus on Saturday July 02, 2022 @11:42AM (#31959971)
Pretty sure it was a good deal more common to do so than it is now.
Re:How the US Air Force Ditched the Average and Saved Lives by Zenst on Saturday July 02, 2022 @10:14AM (#31959155)
Issue I have with averages is they never say how worked out as often they will use a method that suits the outcome they wish. You get different results from mode, medium and mean averages and can cherry pick wich one fits the narrative and use that with the label `average` and it's just accepted by the majority.

Personaly ALL averages should list all three averages for context and clarity as they do offer a greater insight.

Re:How the US Air Force Ditched the Average and Saved Lives by stepanhruda on Saturday July 02, 2022 @10:26AM (#31959245)
Is this actually very common? “average” is the common word for “mean”, I haven’t seen anyone to present median or mode as an average.
Re:How the US Air Force Ditched the Average and Saved Lives by Zenst on Saturday July 02, 2022 @11:05AM (#31959624)
That's the problem as mean can be skewed and sure it is often the go to use for averages but can slant the data.

example:

data 1, 2, 2, 2, 10

Mean: 3.4 Median: 2 Mode: 2

Nice online tool to easily work these out https://www.calculator.net/mean-median-mode-range-calculator...

Re:How the US Air Force Ditched the Average and Saved Lives by stepanhruda on Saturday July 02, 2022 @11:36AM (#31959911)
Yes, it can be a problem and I agree more context like median or a full histogram should be more common, but I have never seen anyone claim that the average was 2 in your example.
Re:How the US Air Force Ditched the Average and Saved Lives by texaslonghorn5 on Saturday July 02, 2022 @03:31PM (#31961651)
While the average value was 3.4, I don't think it would be unfair to suggest the average person would have scored a 2 (or whatever the relevant scenario is).
Re:How the US Air Force Ditched the Average and Saved Lives by kqr on Saturday July 02, 2022 @11:06AM (#31959636)
Depending on the purpose of finding a central tendency of those numbers, I would argue the mean is correct and the others are skewed...
Re:How the US Air Force Ditched the Average and Saved Lives by Zenst on Saturday July 02, 2022 @03:36PM (#31961692)
I was always bemused that the average family size was 2.4 children as no family fits that average ever. With mode and medium that would not happen.

Of note I have no idea what the average family size is children wise, just going on longstanding data from UK that in itself may be out of date, though does highlight the point.

[EDIT fixed typo]

Re:How the US Air Force Ditched the Average and Saved Lives by kqr on Saturday July 02, 2022 @04:48PM (#31962236)
Yet if we set up a game with a penalty proportional to the distance between your guess and the true number of children for a family, and you guess the median or mode, you'll be right more often than I (because I will never be right), but you will also continually be losing money to my guess of the mean.

What matters in real life is generally not how often you are correct, but rather by how much you are wrong. What you need to minimise is not your error rate, but the consequences when you are wrong. You are free to make an infinite amount of mistakes, as long as you are sure to make them in such a way that they are comparatively insignificant.

Re:How the US Air Force Ditched the Average and Saved Lives by Zenst on Saturday July 02, 2022 @05:35PM (#31962671)
> Yet if we set up a game with a penalty proportional to the distance between your guess and the true number of children for a family, and you guess the median or mode, you'll be right more often than I (because I will never be right), but you will also continually be losing money to my guess of the mean.

Not sure were you went there, but the point is that you can overdetail in a way that abstracts from reality.

Re:How the US Air Force Ditched the Average and Saved Lives by kqr on Saturday July 02, 2022 @11:05AM (#31959628)
It depends on your definition of "very common" of course, but I have come across people using "average" to mean both "typical" and "most common" and "middle of the range" and "my preference" and a whole slew of other things.
Re:How the US Air Force Ditched the Average and Saved Lives by brnt on Sunday July 03, 2022 @05:19AM (#31966338)
Mean, mode, median; they're all summaries. People even in science-adjacent fields are still often used to communicating a summary without showing the dsitribution. If possible, I always ask for both mean and median; it's an indication how off-normal the distribution is.

The mean only has meaning for normal distribution, so it's my least favorite summary, because it assumes the most. Imho the normal dist is assumed way too often.

Re:How the US Air Force Ditched the Average and Saved Lives by themitigating on Saturday July 02, 2022 @08:02PM (#31963688)
But isn't the average the useful metric for this question. The question being "If I was born in the 18th century how long would I live". Dying as a baby is a valid outcome.

Maybe the question should be "If I lived past 1 years old what is my average life expectancy?"

Re:How the US Air Force Ditched the Average and Saved Lives by mdp2021 on Saturday July 02, 2022 @10:32AM (#31959301)
It is helpful to say that at first quintile they died at around 10 years of age and at last quintile they died at around 70 (with a median at 33). The average may remain 40, but you get a better idea of the distribution.
Re:How the US Air Force Ditched the Average and Saved Lives by secabeen on Saturday July 02, 2022 @05:22PM (#31962568)
The data point you want to use is the modal age of adult death, but everyone is fixed on the mean age of death, which you correctly note is completely skewed by childhood mortality.
Re:How the US Air Force Ditched the Average and Saved Lives by mrwh on Saturday July 02, 2022 @11:16AM (#31959732)
Another way of looking at it is that's even worse though. Today, it's very rare to have to deal with the death of a child; then, almost everyone had to go through that. And to say, "once you reached" should really be "if you reached".