Data scientists need to learn about significant digits

Re:Data scientists need to learn about significant digits by CrazyStat on Monday December 25, 2023 @06:41PM (#38767416)

I have mixed feelings. On the one hand, yes, people working with data should be more thoughtful about how many digits they report. But I’m not a big fan of the proposed solution (report everything to a small fixed number of significant digits, like two).

Massive computer experiments are common and (often) cheap. If I’m comparing two classification models on a large dataset a difference between 97% and 96.8% accuracy might well be both statistically and practically significant.

An error I’ve seen professional statisticians make is asserting that averaging samples (or performing more complex statistical analyses like regression) justifies increasing the number of significant digits. For example, if I have a hundred observations measured with two significant digits of precision, the mean of those observations is asserted to be accurate to three significant digits, the standard deviation of the mean being 1/10th that of each observation. This is a natural conclusion if you treat the significant digits as accounting only for independent random errors, but in fact significant digits are also used to account for systematic errors in measuring equipment. A scale that measures down to 0.1g, for example, may be miscalibrated by 0.03g. This makes reporting the mean of 100 samples down to 0.01g obviously nonsensical.

In short, how many digits to report deserves more thought than either most data scientists or the author give it.

[ Parent | on HN ]

Re:Data scientists need to learn about significant digits by YetAnotherNick on Monday December 25, 2023 @08:05PM (#38767952)

> 97% and 96.8% accuracy might well be both statistically and practically significant.

I don't think author ever objected to this.

[ Parent | on HN ]

Re:Data scientists need to learn about significant digits by CrazyStat on Monday December 25, 2023 @08:28PM (#38768094)

> Which answer do you give? Whatever your software tells you (e.g., 87.14234%) or a number made of a small and fixed number of significant digits (e.g., 87%).

> The latter is the right answer in almost all instances.

and

> Most human beings are happy with a 1% error margin.

While he doesn't say it as a rule outright, the author repeatedly uses a "small and fixed" two significant digits. He later says this:

> You must choose the number of significant digits deliberately.

which I agree with, but is at odds with the "small and fixed" dictum with which he leads off the post.

[ Parent | on HN ]

Re:Data scientists need to learn about significant digits by YetAnotherNick on Tuesday December 26, 2023 @04:22AM (#38770178)

Fixed doesn't mean 2. If it is statstically and practically significant. It could be 3 or 4.

[ Parent | on HN ]

Re:Data scientists need to learn about significant digits by CrazyStat on Tuesday December 26, 2023 @01:15PM (#38774276)

If you're adjusting the number of significant figures you report based on what is practically and statistically significant then it's no longer fixed.

[ Parent | on HN ]

Re:Data scientists need to learn about significant digits by YetAnotherNick on Wednesday December 27, 2023 @07:22PM (#38788527)

It doesn't need to be same for all the task, just fixed for a particular task. The creators of benchmark or the community could come up with digits of significance that should be the standard, which is based on expected error range. For imagenet it could be 4, for ARC it could be 2.

[ Parent | on HN ]