Re:Demystifying Floating Point Precision
by wruza on Wednesday November 22, 2017 @02:27PM
(#15759688)
Base 2 doesn’t work well with military too:
http://www-users.math.umn.edu/~arnold/disasters/patriot.html
Re:Demystifying Floating Point Precision
by evincarofautumn on Wednesday November 22, 2017 @10:13PM
(#15762469)
Doing calculations with an exact number type such as [big] rationals, and then explicitly converting to floating point or rounding/truncating digits of precision as needed, is a pretty good middle ground for many applications. 109/100 − 109375/100000 = 109/100 − 35/32 = −3/800 = −3.75ᴇ−3.
Real world example: 1.09 - 1.09375
Double: 0.00374999999999992
Decimal: 0.00375