Re:E-Stop and Fuel, software that keeps you awake at night
by _0ffh on Sunday February 11, 2018 @07:22AM
(#16352389)
Yeah, it' simple to see in this example. Though it's often harder to see in practice, I once heard/read some interesting advice: Test your software in all four rounding modes. If the software works correctly (and approximately agrees) in all four modes, you're good. (I suppose this assumes that you've got reasonable test coverage.)
E.g.
Having a tiny step value means substantially worse error due to rounding. Instead of being a tiny fraction of a percent off now you've got something potentially dangerous like 30% off. You can't just handwave away loss of precision as something to ignore, there are a handful of common pitfalls that will amplify the total error to the point where it's unacceptably high and not intuitive to figure out the cause of the error.