News for hackers. Stuff that matters.
Re:Where do Google's flaky tests come from? by brianberns on Saturday February 02, 2019 @12:56AM (#19061402)
If someone showed me that my tests were not deterministic, I'd be in a hurry to figure out why. Hidden race conditions? Unreliable initial state of the system? Unexpected interactions between tests? I'm just surprised that this article doesn't mention any effort to find the root cause and fix it.
Re:Where do Google's flaky tests come from? by SpicyLemonZest on Saturday February 02, 2019 @01:35AM (#19061528)
The challenge isn’t when your tests are flaky. The challenge is when a test written by some guy who left the company begins flaking, and your preliminary investigation shows it’s a timing issue two layers below you in the stack, but the people who own that project insist you’re using it wrong and your test shouldn’t pass in the first place. It’s hard to sell why your team should drop everything to untangle that knot.
Re:Where do Google's flaky tests come from? by tacostakohashi on Saturday February 02, 2019 @08:14AM (#19062814)
These are all just excuses, and common ones at that. Think of it this way:

* The test cases written by the some guy who left the company are now your test cases.

* Before he left the company, he was able to commit the test cases, passing whatever review / quality processes were put in place by the company (or that the company failed to put in place).

* If your team wants to have the benefit of reliable testcases, the knot needs to be untangled. That doesn't entail "dropping everything" (although perhaps thats the only way your organization knows how to operate), there is no reason that improving test cases can't be prioritized alongside and in the context of other work.

Unfortunately, in my experience it's usually the last bullet point where things fall down. Product owners consistently pressure development teams to deliver features now, and do test cases later, which means never.

Re:Where do Google's flaky tests come from? by SpicyLemonZest on Saturday February 02, 2019 @05:33PM (#19066223)
The last bullet point often should fall down. There are a lot of tests that aren’t worth the effort of making them run reliably, but most organizations have (understandably) strong norms against fixing a failing test by removing it.
Re:Where do Google's flaky tests come from? by JoeAltmaier on Saturday February 02, 2019 @08:18AM (#19062831)
Its all about competition. You move fast and deliver rapidly, or get left behind. Time is actually money, for an engineer at least. So unless those tests are needed now, it always seems better to de-prioritize them.
Re:Where do Google's flaky tests come from? by Buge on Saturday February 02, 2019 @04:00AM (#19061981)
Any test with a normally non-hit timeout anywhere is nondeterministic, because the operating system could schedule your test very slowly causing it to hit the timeout.