Why does 0.1 and 0.2 = 0.30000000000000004?

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by jdmichal on Wednesday February 08, 2023 @06:03PM (#34716888)

GP didn't say odd, or at least not numerically odd, but rather "ending in impossible digits".

Floats have varying precision, not uniform. The closer the number is to zero, the more precision it has. There's a point where the precision is so low that it only covers whole numbers, and then a point after that where the precision is less than whole numbers.

Given the context of pointers, it's quite possible that they were large enough to reach that less-than-whole-number-precision range.

EDIT: For 32-bit floats, that's apparently any number above 16,777,216. Which seems surprisingly small to me. For 64-bit doubles that goes to 9,007,199,254,740,992.

https://blog.demofox.org/2017/11/21/floating-point-precision

[ Parent | on HN ]

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by jkaptur on Wednesday February 08, 2023 @07:57PM (#34718205)

I think the comment may have been edited ;)

Anyway, yes, that's all true. Excel, Sheets, JavaScript numbers, Java doubles, Python floats, and so on all work this way. That's why I asked how switching spreadsheet implementations solved the problem.

[ Parent | on HN ]

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by jdmichal on Thursday February 09, 2023 @12:24PM (#34727519)

Ah, sorry for that I must have seen the updated one.

I agree with you that switching programs that are all using the IEEE floating-point formats with the same underlying hardware implementations should result in the same answers.

[ Parent | on HN ]