Re:Formatting floating point numbers
by karmakaze on Friday June 14, 2019 @04:52PM
(#20186585)
It doesn't take much imagination to construct an example. Knowing 0.1 isn't exactly represented make a formula with it that should be exactly a half cent. Depending on fp I precision it will be slightly above or below a half cent and rounding will not work as expected. We found this in prod at a rate of hundreds to thousands of times per day it only takes volume to surface unlikely errors.
[ Parent
| on HN ]
I thought I was clear that the result was rounded. The fp imprecision affects the correctness of rounding e.g. half to nearest even.