News for hackers. Stuff that matters.
Re:999999999999999 - 999999999999997 by swolchok on Monday August 24, 2009 @05:33PM (#783436)
The algorithm probably first decides how to do the math and then does the math.

EDIT: single-precision does behave this way:

    -bash-3.2$ cat precision.c
    #include <stdio.h>
    int main() {
      float f1 = 999999999999999;
      float f2 = 999999999999997;
      printf("%f\n", f1 - f2);
      return 0;
    }
    -bash-3.2$ gcc precision.c
    -bash-3.2$ ./a.out
    0.000000
I am saying that if it's not consistent with single-precision across multiple inputs, there's probably code that looks at the numbers involved and decides what kind of numeric type to use to do the calculations. In this case, single-precision floating-point happens to be a bad choice.

Second edit: replacing the 7 with a 6 still prints 0. The parent is right: something else is going on here. I like the theory espoused elsewhere in this discussion that it's a result of some truncation to avoid returning nonzero when the result should be zero.

Re:999999999999999 - 999999999999997 by gjm11 on Monday August 24, 2009 @06:55PM (#783604)
What exactly do you mean, and how does it make "using floating point arithmetic with insufficient precision" a viable explanation for this, given that none of the usual floating-point formats behave in this way?