Re:Why does 0.1 and 0.2 = 0.30000000000000004?
by mananaysiempre on Wednesday February 08, 2023 @10:29PM
(#34719399)
Excel uses floats in its own ways that even experts on floating-point arithmetic find inscrutable[1].
[1] https://people.eecs.berkeley.edu/~wkahan/Mindless.pdf, §2
Re:Why does 0.1 and 0.2 = 0.30000000000000004?
by t8sr on Thursday February 09, 2023 @04:20AM
(#34721518)
Ah, but it used ints (or bignum) to represent my pointer values, instead of silently losing precision. This is an anecdote, maybe there are other gotchas I am blissfully ignorant of. (Also, this was 8 years ago.)
But Excel also uses floats (as in, Single-precision floating-point format, aka float32), because of "compatibility with Lotus 1-2-3".
https://softwarerecs.stackexchange.com/questions/53292/any-s...
https://learn.microsoft.com/en-us/office/troubleshoot/excel/...