News for hackers. Stuff that matters.
Re:The most surprising Unix programs by nullc on Sunday March 15, 2020 @02:52PM (#22585666)
It's also a performance optimization though, since otherwise would might instead just use 400 digits of precision (or whatever) all the time, and round just the output.
Re:The most surprising Unix programs by bloomer on Sunday March 15, 2020 @03:47PM (#22586109)
Somewhat, in that it doesn’t use more precision than needed, but the real issue is you can’t just pick some arbitrarily large precision and round at the end. For some calculations, even 400 digits during intermediate steps would not be enough due to catastrophic cancellation and you would need to go even higher precision to get the right answer. It really is about solving an accuracy issue and not an optimization. And determining that you are using sufficient precision to get an accurate answer is an extra cost, so it is always more expensive than just plowing ahead and calculating an inaccurate answer.
Re:The most surprising Unix programs by nullc on Sunday March 15, 2020 @05:50PM (#22587120)
Presumably there is a maximum precision or otherwise a seemingly innocuous calculation could run you out of memory.