News for hackers. Stuff that matters.
Re:Formatting floating point numbers by civility on Friday June 14, 2019 @05:15PM (#20186728)
The person I replied to claims to have looked at large* data sets and never seen a discrepancy in 6-8 significant digits. I thought I'd show them a small data set with 3 samples that retains no significant digits.

* Never mind that "millions" isn't large by current standards...

Re:Formatting floating point numbers by firethief on Friday June 14, 2019 @05:57PM (#20186986)
But you are observing all the digits, not just 6-8. It's implicit in the semantics of the operation, and that's something everyone who works with floating point should know.
Re:Formatting floating point numbers by civility on Friday June 14, 2019 @10:04PM (#20188025)
I think I see where you're hung up. If I change it to:

    x = 1e20 + 1 - 1e20
    y = 1e20 - 1e20 + 1
    assert x == y
There is only 1 digit, and it's wrong. You don't even need 6-8. I probably should've used this as my example in the first place.
Re:Formatting floating point numbers by firethief on Saturday June 15, 2019 @11:18AM (#20190379)
You're making the same mistake but now it's less obvious because the change of scale. When you compare floating point numbers, simple == is not usually what you want; you need to compare them with a tolerance. Choosing the tolerance can be difficult, but in general when working with small numbers you need a small value and with large numbers you need a large value. This dataset involves datapoint(s) at 1e20; at that magnitude, whatever you're measuring, the error in your measurements is going to be way more than 1, so a choice of tolerance ≤ 1 is a mistake.
Re:Formatting floating point numbers by civility on Saturday June 15, 2019 @04:26PM (#20191928)
Ugh, you're preaching to the choir. I wasn't trying to make a point about the equality operator, I was trying to make a point about x and y being completely different. I must be really bad at communicating with people.
Re:Formatting floating point numbers by firethief on Saturday June 15, 2019 @07:03PM (#20192728)
That's the thing: x and y are not "completely different"; they are within the tolerance one would use to compare them correctly.
Re:Formatting floating point numbers by civility on Saturday June 15, 2019 @09:22PM (#20193245)
So umm, more than a "difference in the realm of 6-8 significant digits"?
Re:Formatting floating point numbers by firethief on Sunday June 16, 2019 @01:55PM (#20196863)
That construction can turn any residual N digits out into a first digit difference. It wouldn't matter without making comparisons with tolerance way under the noise floor of the dataset. But yes, you have technically invented a situation that differs from that real-world anecdote in regard to that property, in an extremely literal interpretation.
Re:Formatting floating point numbers by civility on Monday June 17, 2019 @12:53PM (#20204896)
And here I was, worried I might be tedious or pedantic, trying to argue that floating point is just not that simple. You've really outdone me in that regard.