News for hackers. Stuff that matters.
Re:TTNT: Test This, Not That by wdewind on Friday October 02, 2015 @10:21AM (#10318703)
I remember reading the original article that inspired it and thinking "huh, that's a cool idea." In practice I wonder if they'll ever really sort this out:

> Test selection algorithm is not perfect yet (it may produce false-positives and false-negatives)

The problem is you're going to need to run the entire test suite before deploying anyway unless the algorithm is perfected. If you have to do this anyway, while it's helpful to have more confidence going into an hour long test build that it's going to pass, it's probably more helpful to just build an infrastructure on which the entire test suite runs more quickly. I've worked on sites with extremely large test suites and it was never an issue to run the tests in < 10 minutes because they could be run on an infrastructure that allowed tests to be run in parallel. I think that's the solution rather than a complex and flakey algorithm.

Re:TTNT: Test This, Not That by mannykannot on Friday October 02, 2015 @01:32PM (#10319965)
> Test selection algorithm is not perfect yet (it may produce false-positives and false-negatives)

I haven't read the code, but the associated document explaining the method makes no mention of accounting for line renumbering as a result of changes.

The accuracy with which changes are identified depends on the diff algorithm. I imagine it is more likely to generate false positives than false negatives (which is preferable), but I am sure there are cases where a line is unchanged textually but is different semantically (e.g. by the insertion or removal of an 'else' above it), and the diff program will consider it unchanged.

Re:TTNT: Test This, Not That by wdewind on Friday October 02, 2015 @03:32PM (#10320720)
It's not only that, once you know whether or not something has changed you then need to know what tests need to be run.
Re:TTNT: Test This, Not That by __alexs on Friday October 02, 2015 @10:58AM (#10318954)
I implemented much the same thing for Python a few years ago. Accuracy on real programs is actually quite good but ultimately Python and Ruby don't really make it easy to get accurate results. This is mostly caused by (perhaps unsurprisingly) issues around global and otherwise shared state between tests that is often difficult to handle efficiently.

In my prototype this caused effectively any change outside of a function body to end up disabling the optimisation entirely :(

Re:TTNT: Test This, Not That by wdewind on Friday October 02, 2015 @03:33PM (#10320737)
Why would the language being untyped have anything to do with whether or not there is global state? I think I'm missing something.
Re:TTNT: Test This, Not That by hinkley on Friday October 02, 2015 @02:43PM (#10320432)
Flakey always seems tolerable when you have 400 tests and you get one or two false reports a week. What people don't get is the the chance of failure is multiplicative, not additive, not fixed. So when you use that function 10 times as often it fails 12 or more times as often.

Tests are there to fight our natural tendency toward wishful thinking. Tests that fail for no reason enable people to keep thinking there was nothing wrong with their code. Meanwhile other people are twiddling waiting for them to fix something they don't think they broke.

Re:TTNT: Test This, Not That by wdewind on Friday October 02, 2015 @03:30PM (#10320706)
Sorry I didn't mean flakey in the sense of the tests themselves, but the algorithm that selects which tests need to be run.
Re:TTNT: Test This, Not That by stupidcar on Friday October 02, 2015 @12:16PM (#10319432)
It seems like there could be more value than just speeding up test runs though. Imagine a system that gave feedback, just prior to committing, about the number of tests that your changes will trigger. If the number of tests is larger than you intuitively expect, it could be an indication that the change has side-effects you haven't anticipated.