Why does 0.1 and 0.2 = 0.30000000000000004?

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by radu_floricica on Thursday February 09, 2023 @02:31AM (#34720812)

something like, in mysql: create table t ( value double(10,2) not null ); put a bunch of values, something like 15k with maybe half of them being 0.37.

Do the sum two ways:

- export them in excel and do a sum

- select sum(value) from t

and you get two different values, with a difference of something like 15.

Replace double(10,2) with decimal(10,2), and the sum(value) works correctly.

I still have no idea why this happens. I could expect a tiny difference, and code like your above will indeed give differences like this, well under 1.0, but my particular scenario had differences many orders of magnitude higher. Still can't explain.

[ Parent | on HN ]

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by archi42 on Thursday February 09, 2023 @04:21AM (#34721533)

something like 15k with maybe half of them being 0.37

Oh, that's something entirely different. If some of your other numbers are much larger than 0.37, you will run into a different kind of problem.

You know about exponent and mantissa? Adding a value with a small exponent to a value with a big exponent will cause imprecisions. In extreme cases the result is just the bigger number.

[ Parent | on HN ]