News for hackers. Stuff that matters.
Re:How we took our tests from 100 seconds to 100ms by hinkley on Tuesday July 21, 2015 @04:05PM (#9925343)
Someone who checks in broken code will also take flaky tests as an excuse not to look at the build results, or to convince themselves that the failure is caused by flakiness and not their changes.

If I had a dollar for every time I asked someone why they weren't fixing their shit, and they answered "Oh, I thought the build was failing randomly again." I'd have a lot of dollars.

Re:How we took our tests from 100 seconds to 100ms by phpnode on Tuesday July 21, 2015 @04:24PM (#9925510)
The fact that your tests are flaky at all means that the integration tests have revealed a bug that you need to fix. The tests have done their job. The fact that people aren't looking at the results speaks to a larger cultural issue.
Re:How we took our tests from 100 seconds to 100ms by hinkley on Wednesday July 22, 2015 @01:18AM (#9927863)
Not if it's a timing error in the test harness. That's usually what people mean when they say flaky tests. Selenium, especially three or four years ago, would simply fail occasionally no matter how good your tests are.

Yes, there are bugs that look like bad luck, but are really flaws in your logic. I have fixed a lot of bugs that look Random, but eventually you're left with, for instance, selenium just refusing to click a button.