Re:One does not simply calculate the absolute value
by pedrocr on Sunday August 22, 2021 @04:14PM
(#28269151)
For double it would be harder to check but for float (that should have the same corner cases) it's easy to just check all possible values if you have a gold standard. I've been doing that for image processing functions and it's amazing how many corner cases you fix and keep fixed by writing unit tests that iterate over all possible values or a good subset of them when that's not possible.
[ Parent
| on HN ]
Re:One does not simply calculate the absolute value
by Strilanc on Sunday August 22, 2021 @07:06PM
(#28270584)
I second "just try all of them" when testing single precision floats. It's really easy to do, and gives you a lot of confidence in the correctness. For example, I had a float-to-bytes-to-floats conversion in some GPU shaders and "just test that every single possible float survives a round trip" made it trivial to be sure it was correct. (Also I found out my video card drivers were losing an ulp when parsing float constants in shaders.)
Re:One does not simply calculate the absolute value
by pedrocr on Monday August 23, 2021 @05:45AM
(#28273832)
Roundtrip tests are great when you don't have a gold standard. It allows for at least consistency across operations. I've done a bunch and found quite a few corner cases. I also did a few roundtrip tests in RGB spaces where being exhaustive was no longer feasible. For those I ended up iterating over the whole RGB space but skipping values in each component by different prime values, to try and cover as many of the possible weird cases as possible:
https://github.com/pedrocr/imagepipe/blob/12269f04ac2fa0d165...