News for hackers. Stuff that matters.
Re:Why does 0.1 and 0.2 = 0.30000000000000004? by radu_floricica on Wednesday February 08, 2023 @01:57PM (#34712929)
I thought that until literally two days ago. Turns out that if you sum a bunch of 0.37 (not even huge numbers, just around a few thousand) you end up with differences on the order of 10-20. Both in mysql and in Java. No, this doesn't makes sense to me either - the differences should be a LOT farther than 3 digits. And yet.

You should have seen my face when debugging this.

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by zeven7 on Wednesday February 08, 2023 @02:16PM (#34713281)
I'm curious about this. Could you provide an example?

Here's what I'm seeing:

---

JavaScript

    (function() {
      let inc = 0.37;
      let times = 5000;
      let total = 0;
      for (let i = 0; i < 5000; i++) {
        total += inc;
      }
      console.log('expected:', inc \* times);
      console.log('actual:', total);
    })();
expected: 1850

actual: 1849.9999999997679

---

Java

    public class MyClass {
        public static void main(String args[]) {
            double inc = 0.37;
            double times = 5000;
            double total = 0;
            for (double i = 0; i < 5000; i++) {
                total += inc;
            }
            System.out.println("expected: " + (inc \* times));
            System.out.println("actual: " + total);
        }
    }
expected: 1850.0

actual: 1849.9999999997679

---

update: Changing `double` to `float` in Java yields:

expected: 1850.0

actual: 1849.9778

and maybe that lines up with what you meant by "the differences should be a LOT farther than 3 digits", though it's hard to tell what you mean by "differences on the order of 10-20".

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by HNDen21 on Wednesday February 08, 2023 @03:37PM (#34714662)
same in SQL Server

   declare @i float =0.37,@times int = 0,@total float = 0

   while @times < 5000
   begin
   set  @total+=  @i
   set @times +=1
   end

   select @total
1849.99999999977

or

   select sum(t) from(
   select top 5000 convert(float, 0.37) t 
   from sysobjects a cross join sysobjects b) z
1849.99999999977
Re:Why does 0.1 and 0.2 = 0.30000000000000004? by radu_floricica on Thursday February 09, 2023 @02:31AM (#34720812)
something like, in mysql: create table t ( value double(10,2) not null ); put a bunch of values, something like 15k with maybe half of them being 0.37.

Do the sum two ways:

- export them in excel and do a sum

- select sum(value) from t

and you get two different values, with a difference of something like 15.

Replace double(10,2) with decimal(10,2), and the sum(value) works correctly.

I still have no idea why this happens. I could expect a tiny difference, and code like your above will indeed give differences like this, well under 1.0, but my particular scenario had differences many orders of magnitude higher. Still can't explain.

Re:Why does 0.1 and 0.2 = 0.30000000000000004? by archi42 on Thursday February 09, 2023 @04:21AM (#34721533)
something like 15k with maybe half of them being 0.37

Oh, that's something entirely different. If some of your other numbers are much larger than 0.37, you will run into a different kind of problem.

You know about exponent and mantissa? Adding a value with a small exponent to a value with a big exponent will cause imprecisions. In extreme cases the result is just the bigger number.