Re:Floating point error is the least of my worries (2011)
by Someone on Thursday February 22, 2018 @03:50AM
(#16436217)
”with a correct workaround which is rounding (for finance) and an horrible evil twin "formatting away and praying" ”
Are you suggesting the “precision as displayed” setting in Excel is that evil twin? Why?
Re:Floating point error is the least of my worries (2011)
by Twisell on Thursday February 22, 2018 @04:22PM
(#16441511)
Because financial data (in our use case) have a fixed 2 decimal format. In almost every other context you could impose this constraint by somehow defining a data type and be done with it.
Real financial data manage to produce three floats (with only 2 decimal places as usual with finance) that when used together a+b-c produced -0 (actually -0.1336e-20 or something like that).
The result ended up producing a cascade of mismatching results in further calculations seeking for 0 balance.
It’s a known and identified "feature" with a correct workaround which is rounding (for finance) and an horrible evil twin "formatting away and praying" that can lead to futur unfigurable mess for candid users.
So for me I agree that floating point error is not so bad from a mathematical standpoint, but from a technical stand point let us all remind that for most humankind 3/2 = 1.5 (and not 0).