Re:Spark joy by running fewer tests
by smarterclayton on Thursday June 11, 2020 @06:20PM
(#23493875)
Of course, to be able to see the signal, you have to be ruthless to noise. Deflake first, disable if you must, but never stop until it goes green and then be ruthless about keeping it that way.
One of the culture changes I've had to try to push was to test for the failure rate of a distributed system rather than the absolute number of failures- don't test for "Were number of failures less than n", test for "Did the success rate fall below 99.99%"
I totally understand, in our setup, flaky tests happen when testing parts of the software that depend on the Hadoop infrastructure. Sometimes you can’t fully control some pieces and as a result those tests can be flaky.
However I want to highlight how frustrating it is when your tests fail when you wait for the build to finish...
For instance in Kubernetes we run lots of tests that verify parts of the system actually working together, not just mocked. A fair amount of the time, once tests are in and soaked, the test flake is actually the canary that says things like “no, the kernel regressed networking AGAIN” or “no, your retry failure means that etcd lost quorum - which shouldn’t have happened, what on earth is going on, oh, someone suddenly started hotlooping and now the api server is performing 10k writes a second”.
At some point, testing the system in isolation is not enough because all failures are interactions of multiple components. Your flakes are the evidence that the gigantic stack of software known as “modern computing run from HEAD lol” is full bugs.
It’s definitely a culture change - once anyone anywhere thinks “oh it’s just flaky” they stop treating it like signal. Once they treat it like noise, it’s very hard to unwind.
Today, when I look at one of those test suites and see a flake, I assume that everything in open source is broken, again, and pull out the shovel, because 75% of the time it is.