News for hackers. Stuff that matters.
Re:Coverage is not strongly correlated with test suite effec... by thaumasiotes on Monday September 27, 2021 @08:47PM (#28678003)
> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input?

How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input.

Now, if your code is just pumping data from one point to another point, it's possible that the destination will vary its behavior based on the input in a way that your coverage-based testing can't see. But if you're doing something with the input yourself, then 100% coverage means you tested every possible kind of input.

(Actually, a "branching" problem can still arise if you have something like

  if( condition1 ) {
    do_stuff();
  }
  
  if( condition2 ) {
    do_other_stuff();
  }
  
  if( condition3 ) {
    do_other_other_stuff();
  }
Because with this code, it's possible to get test coverage of every line of code (there are 6 of them, not counting the '}'s) while not covering every branch (there are 8 of them, depending on the combined truth values of condition1, condition2, and condition3).)
Re:Coverage is not strongly correlated with test suite effec... by spankalee on Monday September 27, 2021 @09:01PM (#28678097)
That's because you have such simple conditions. How do you compute them?

   if (x < y) { ... }
really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows?

Expressions can have any number of edge cases that code coverage can't account for.

Re:Coverage is not strongly correlated with test suite effec... by thaumasiotes on Tuesday September 28, 2021 @04:22AM (#28680459)
Well, imagine two different analyses of the code:

1. Does each branch do the right thing?

2. Does the code go down the correct branch in the first place?

I've focused more on the first question, and you're focusing more on the second. But it's fair to say that each question is a reasonable focus of testing, which weakens my comment above.

Re:Coverage is not strongly correlated with test suite effec... by Supermancho on Monday September 27, 2021 @10:19PM (#28678583)
Depends on the language/type safety offered. If your language has a NaN (javascript) and you can assert the expected behavior of x < NaN (false) or if you don't even want to encounter that, then you're able to reason back to the generation of the inputs. Wherever x and y came from, if there was a possibility of an NaN being a value, you want the generating function to exception or cast via some sort of validation before handing them off to other unsuspecting functions.
Re:Coverage is not strongly correlated with test suite effec... by Supermancho on Monday September 27, 2021 @11:29PM (#28679000)
In my experience, when a function has more than 1 if statement, it's probably hard to test and you might want to split it up.

When a function has has more than 2 if statements, you definitely want to break it up.

Imagine the function where conditions aren't related at all, as you posited:

    myfn(int x, int y, string foo) {
        if(foo == "bar") {
            do_stuff();
        }
  
        if( x < y ) {
            do_other_stuff();
        }
  
        if(time() == "noon") {
            do_other_other_stuff();
        }
    }
What's really happening here? Why is all this wrapped in 1 function, when the args aren't related at all, nor the work they are dependent on?

Let's say the caller was doing: myfn(1,2,foo);

I would split this function into 3 different function calls in the caller...

    possiblyDoStuff(foo);
    possiblyDoOtherStuff(1,2);
    possiblyDoOtherOtherStuff(time());
Let's use something that you wouldn't simply refactor upstream:

    myfn(string go, string for, string foo) {
        if(go == foo) {
            do_stuff();
        }
  
        if(for == go) {
            do_other_stuff();
        }
  
        if(foo == for) {
            do_other_other_stuff();
        }

        someOtherFn(go, for, foo);
    }
Move that complexity into smaller chunks:

    myfn(string go, string for, string foo) {
        possiblyDoStuff(go, foo);
        possiblyDoOtherStuff(go, for);
        possiblyDoOtherOtherStuff(for, foo);
        someOtherFn(go, for, foo);
    }
Now you have the 7 tests. 2 for each possibly = 6 which are pretty easy and 1 for myfn. It is exceedingly rare that functions require this kind of attention. There are usually a bunch of side effects or return values that are dependent on these checks. eg:

    myfn(string go, string for, string foo) {
        // examples of why you are calling these, to get vals
        var go2 = possiblyDoStuff(go, foo);
        var for2 = possiblyDoOtherStuff(go, for);
        var foo2 = possiblyDoOtherOtherStuff(for, foo);
        return { go2, for2, foo2 };

        //move this to caller someOtherFn(myfn(go, for, foo));
    }
Re:Coverage is not strongly correlated with test suite effec... by strken on Monday September 27, 2021 @09:09PM (#28678142)
This isn't true at all. Consider buffer overflows, bugs in your bitwise arithmetic, division by zero, or null pointer exceptions.