Cool idea.. Has this strategy already been tried before?

Facebook and Google employ AI based methods for skipping tests (See): https://static.googleusercontent.com/media/research.google.c...

they are pretty much flawed in a way, because they require you to wait weeks/months to gain statistical sagnificance. And even then they are too costly to train to make sense in smaller scales.

There's an old ex-Google employee who came out with something similar in 2005 for JUnit java tests. We talked to him and ulitmatly he sunsetted the project because static analysis back then was basically copying language parser code from the Language distro and retro fitting it.

Static analysis has gone a long way since though... and no one has yet to deploy something that works for system tests and integration tests in this strategy.

How it works: First time you run nabaz:

1. Collects code coverage for every tests that runs, seperatly.

2. Saves code coverage in local sqlite db, along with commit ID as key.

Second to n runs later:

1. Traverse git tree and queries db via commit ID -> per test code coverage

2.a git diff between new code (code change) and the code coverage from the old commit, test case by test case.

2.b. If an intersection between new code and old code coverage is found, test is rerun. else, it will be skipped.

How do you deal with tests that interact with something external to the codebase? The result of those could potentially change without a code change.

That's a great point!

Depends on the what you define by "externality".

I like to look at it this way, what's being tested? I see three groups:

  - testing the software you wrote.
  - testing integration with other software you wrote (microservice interaction maybe?)
  - testing integration to third party software.

I'm telling you our current plan, feel free to poke holes:

In the first case the only externalities that seem to really matter are mostly resource and configuration files. We could detect I/O with such files and rerun the test if the underlying resource/config changed.

In the second case were planning on collecting coverage for all services being tested and then employ a more sophisticated diff heurastic.

For the third case you can mark the test as always run

We have a poc of a syscall tracer that gives us an I/O map of the test being run or the software being tested. this includes reading/writing to files, pipes, network etc...

This sounds like the best you could possibly do!

What are the odds this baby will be made compatible with Java?

java is scheduled next, would absolutely love to hear about your use-case?

I have a pretty large Java code base (40+ micro services) and the integration test suite takes hours, many hours, to run.

However they are usually testing components from end-to-end. This seems not very amenable to benefitting this approach, since "side effects" are an inherent part of nearly every integration test.

Wow! Seems like a challenge to test, would love to see what we can do.

Regarding side effects, completely fine as long as they're mapped out. Would love to hear more, nothing pitchy or salesy.

hmu [email protected]

This looks interesting and definitely something missing from our CI pipeline. What languages do you intend to support?

Basically all of them as long as their used.

Would love to know more about your use case, email is somewhere in the comments

Sounds super cool! I’ll definitely show this to our devops guy in the company. However sounds like there might be a lot of edge cases that don’t include any code changes, e.g. some environment variable changes - have you thought about it?

Hey, awsome! feel free to contact us at [email protected].

About the env vars: yes. We have a list of edge cases that might affect the tests. Env vars specifically is the next edge case we are about to cover.

Roey from nabaz.

THIS point though, Im inclined to share our edge case doc with HN and let people add stuff.

Thanks guys for all your feedback!

We took all notes to heart, the main problem we identified that currently (at this stage of the product cycle we don't yet support all edge cases)

And this hurts usability and value, will figure out a way to deliver this kind of tech in a way that is still valuable at this early stage :)

Yuval

I'm wondering if this works similar to https://github.com/digitalocean/gta which will do transitive analysis to find packages whose dependencies have changed and only run the tests for those packages?

We love gta! A few key differences though: 1. We analyze the effect of code changed on code coverage, which means we only compare changed code to runtime information (code that actually ran). gta analyzes statically.

Let's assume you have a function:

  func myFunction(a int) {

      if(a > 10) {  

          func1()  

      } else if (a < 4) {  

          func2()  

      } else {  

          func3()

      }  
  }

let's assume you changed func3 only, or func2, but during the run of the test only func1 ran.

The test will be skipped because no change could of affected it's result. NOT in gta.