News for hackers. Stuff that matters.
Re:Can you safely parse a double when you need a float? by im3w1l on Wednesday December 01, 2021 @07:50AM (#29403181)
> With floating-point numbers, I get this effect with the string “0.004221370676532388” (for example). When I round it directly to a float, I get 0.0042213709093630313873291015625. If I first round it to a double and then to a float, I get 0.0042213709093630313873291015625. You probably cannot tell unless you are a machine, but parsing directly to a float is 2e-7 % more accurate.

I scratched my head at this as those are literally the same number.

Looking into it a bit, the correctly rounded float should be 0.0042213709 and the incorrect, twice-rounded is 0.0042213704.

To see what is going on the number in question is in binary base

  0.00000001000101001010011011011001000000000000000000000000000001...
  Rounded to double is
  0.000000010001010010100110110110010000000000000000000000000000
  And when that is rounded to float (round to even mode) you get
  0.0000000100010100101001101101100
  While the correct number is
  0.0000000100010100101001101101101

There are no comments on this comment yet.