Re:Where do Google's flaky tests come from?
by sebcat on Saturday February 02, 2019 @04:43AM
(#19062109)
When doing anything concurrently and/or in parallel, it is always good to have an input queue and an upper bound on the number of tasks (/threads/processes/...). Also an asynchronous flow.
You need to handle other problems (back-pressure, priorities, timeouts, ...) but if you need it you need it.
Re:Where do Google's flaky tests come from?
by zenexer on Saturday February 02, 2019 @06:15AM
(#19062443)
But how do you go about creating tests for all of that?
Re:Where do Google's flaky tests come from?
by cyphar on Saturday February 02, 2019 @08:22AM
(#19062848)
You don't need to write your own and test it -- use something like GNU parallel (if your tests can be run individually as commands or shell scripts) or a test framework that supports parallelism.
Re:Where do Google's flaky tests come from?
by sorokod on Saturday February 02, 2019 @01:38PM
(#19064746)
In such cases it is better to await on some outcome of the asynchronous calculation. In Java ecosystem we have awaitility (https://github.com/awaitility/awaitility) which provides a bunch of utility methods in this spirit.
Re:Where do Google's flaky tests come from?
by slavik81 on Saturday February 02, 2019 @06:28PM
(#19066564)
That's how our tests worked, but you still must choose a timeout or you will wait forever when you do not receive the response you're listening for.
Setting up the CI system to report failure on the first error would have helped a lot too. Waiting on a single timeout was not too bad. The problem was waiting on hundreds of them.
Unfortunately, under high load, I found that threads can be starved of CPU for seconds at a time. The CI machine was flooring it trying to build the software and run the tests. Setting the timeout high enough that there were no random failures caused by CPU starvation meant that build failures would take a very long time, delaying feedback for errors. That was nearly as bad as the flakiness.
I mostly fixed the flakiness by making the number of threads configurable and only running one thread in the testing environment, but I think a better way would be to reduce the number of parallel jobs to keep the test machines' CPU utilization rate at something more reasonable. Then, you can set timeouts that reflect real performance expectations for your software.