Re:Windows Calculator invalid input with 10^x function
by rspeer on Thursday August 02, 2018 @06:36PM
(#17676049)
I wonder if they're using a fraction type to store decimals (a reasonable way to avoid floating-point error), and when the 10^x function is implemented naively on this type, 10^(3.1415) becomes 10^(31415/10000) = (10^31415 / 10^10000) = (overflow / overflow).
Re:Windows Calculator invalid input with 10^x function
by istjohn on Thursday August 02, 2018 @09:34PM
(#17676733)
> 10^(31415/10000) = (10^31415 / 10^10000)
If n^(a/b) = (n^a)/(n^b) then 2^(1/2) = (2^1)/(2^2) = 2/4 = 0.5 but 2^(1/2) = sqrt(2) = 1.4...
Re:Windows Calculator invalid input with 10^x function
by rspeer on Friday August 03, 2018 @02:35AM
(#17677691)
Dang, you're right, I should have known that. I apologize for my bad math.
It would be the 10000th root of 10^31415, and at this point what I'm saying is probably entirely unrelated to how it's really implemented.
My second guess is they are using some weird algorithm that hits an error case due to the limits of whatever number type they are using.