Re:999999999999999 - 999999999999997
by dreish on Monday August 24, 2009 @05:07PM
(#783360)
Except that neither double nor single precision IEEE floats behave the same way as whatever numbers Google is using.
[ Parent
| on HN ]
The algorithm probably first decides how to do the math and then does the math.
EDIT: single-precision does behave this way:
-bash-3.2$ cat precision.c
#include <stdio.h>
int main() {
float f1 = 999999999999999;
float f2 = 999999999999997;
printf("%f\n", f1 - f2);
return 0;
}
-bash-3.2$ gcc precision.c
-bash-3.2$ ./a.out
0.000000
I am saying that if it's not consistent with single-precision across multiple inputs, there's probably code that looks at the numbers involved and decides what kind of numeric type to use to do the calculations. In this case, single-precision floating-point happens to be a bad choice.Second edit: replacing the 7 with a 6 still prints 0. The parent is right: something else is going on here. I like the theory espoused elsewhere in this discussion that it's a result of some truncation to avoid returning nonzero when the result should be zero.
What exactly do you mean, and how does it make "using floating point arithmetic with insufficient precision" a viable explanation for this, given that none of the usual floating-point formats behave in this way?