Re:Formatting floating point numbers
by tomxor on Friday June 14, 2019 @11:17AM
(#20183675)
> in the realm of 6-8 significant digits I looked at
That is far inside the range of 64bit double precision. For error to propagate up to that range of significance depends on the math, but i doubt the aggregation you are describing would cause it... provided nothing silly happens to subtotals like intermediate rounding to precision (you'd be surprised).
Something like compounding as the parent was describing are far more prone to significant error propagation.
That is far inside the range of 64bit double precision. For error to propagate up to that range of significance depends on the math, but i doubt the aggregation you are describing would cause it... provided nothing silly happens to subtotals like intermediate rounding to precision (you'd be surprised).
Something like compounding as the parent was describing are far more prone to significant error propagation.