News for hackers. Stuff that matters.
Re:Formatting floating point numbers by karmakaze on Friday June 14, 2019 @04:52PM (#20186585)
It doesn't take much imagination to construct an example. Knowing 0.1 isn't exactly represented make a formula with it that should be exactly a half cent. Depending on fp I precision it will be slightly above or below a half cent and rounding will not work as expected. We found this in prod at a rate of hundreds to thousands of times per day it only takes volume to surface unlikely errors.
Re:Formatting floating point numbers by karmakaze on Friday June 14, 2019 @05:04PM (#20186663)
I thought I was clear that the result was rounded. The fp imprecision affects the correctness of rounding e.g. half to nearest even.
Re:Formatting floating point numbers by Demiurge on Friday June 14, 2019 @05:00PM (#20186636)
It is the rounding that seems to be an issue here.