Re:Excel_to_code – Converts some Excel spreadsheets to C
by ma2rten on Friday July 24, 2015 @11:39PM
(#9946451)
I tried 0.1 + 0.2 in both google docs and excel. I got 0.3000000000000000000, not 0.30000000000000004, like you normally would using floating point.
[ Parent
| on HN ]
Re:Excel_to_code – Converts some Excel spreadsheets to C
by teraflop on Saturday July 25, 2015 @01:38AM
(#9946684)
I think there's internal rounding that hides it. Try A1=1e50, B1=A1+1, and C1=B1-A1.
Excel definitely used floating point in the past (because you used to be able to use it to test for the Pentium FDIV bug) and according to the documentation, it still does: https://support.microsoft.com/en-us/kb/78113
Re:Excel_to_code – Converts some Excel spreadsheets to C
by timrobinson on Saturday July 25, 2015 @03:05AM
(#9946824)
Rounding happens when the value is displayed. To see the raw value, try: =(0.1+0.2)