I generally agree, but there's one thing that bothers me about the practice:

Before disabling the test, does anyone even consider the possibility that maybe it's not the test that's flaky, but that the software under test is itself buggy in a non-deterministic way?

I don't have a good answer for how to deal with this. It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases. Maybe the heuristic of assuming that it's the test's fault is good enough, and this is what I've done in every team I've worked in.

But it does bother me.

There’s one particular domain where this question runs deep - tests that verify parts of distributed systems. Note: please do not extrapolate any problem described below outside of distributed systems.

For instance in Kubernetes we run lots of tests that verify parts of the system actually working together, not just mocked. A fair amount of the time, once tests are in and soaked, the test flake is actually the canary that says things like “no, the kernel regressed networking AGAIN” or “no, your retry failure means that etcd lost quorum - which shouldn’t have happened, what on earth is going on, oh, someone suddenly started hotlooping and now the api server is performing 10k writes a second”.

At some point, testing the system in isolation is not enough because all failures are interactions of multiple components. Your flakes are the evidence that the gigantic stack of software known as “modern computing run from HEAD lol” is full bugs.

It’s definitely a culture change - once anyone anywhere thinks “oh it’s just flaky” they stop treating it like signal. Once they treat it like noise, it’s very hard to unwind.

Today, when I look at one of those test suites and see a flake, I assume that everything in open source is broken, again, and pull out the shovel, because 75% of the time it is.

Of course, to be able to see the signal, you have to be ruthless to noise. Deflake first, disable if you must, but never stop until it goes green and then be ruthless about keeping it that way.

One of the culture changes I've had to try to push was to test for the failure rate of a distributed system rather than the absolute number of failures- don't test for "Were number of failures less than n", test for "Did the success rate fall below 99.99%"

I totally understand, in our setup, flaky tests happen when testing parts of the software that depend on the Hadoop infrastructure. Sometimes you can’t fully control some pieces and as a result those tests can be flaky.

However I want to highlight how frustrating it is when your tests fail when you wait for the build to finish...

> It would be an enormous amount of effort to go through every flaky result and reason about whether the problem is with the test or the code under test, especially in larger codebases.

IMO a "flaky test" is one for which the investigation has been performed and the design of the test assessed to be "flaky." i.e. not a test that passes some times and not others, but one whose design cannot reliably expect a given result or one whose implementation can be impacted by factors outside of the control of the test suite. Timeouts are a classic source of flaky tests.

Once a test is determined to be flaky it should be evicted from the test suite, or at least demoted from the "reliable/frequently run" rank to a lower one that can tolerate some hand holding.

I'm not sure if it's exhaustive, but any test with a dynamic dependency is one I would likely classify as "flaky", like anything hitting another system. Draw your own boundaries as fit around what "dynamic" means (ex: some might say static files are OK, others not).

Also, I'm talking every-commit, unit-level testing for context.

This is tricky but when you reframe it it's clear the heuristic you mention is the correct one. The value of a test is giving signal to a developer that the code they wrote is correct. If a test is flaky due to the test or the code, the signal the test provides has now become negative. The parent mentions filing it on the team that owns the test. This sounds correct and the onus is then on that team. At larger companies this process is automated. You may enjoy the flake detection section in this blog post https://dropbox.tech/infrastructure/athena-our-automated-bui...

I have found two cases that occur enough to not assume that the test is wrong:

1. latency/performance - the test system in a well-controlled environment shouldn't have so much latency variance anyway (project was an 'instant' messenger)

2. maybe 30% of the time the test (run frequently in CI) is identifying a race condition that doesn't show up when run ad-hoc by a dev

I've always been curious about this. My fiction is that if someone spent the time they'd find a few sources of most of the flakiness and and at some point the flakiness would fall drastically. Like maybe there are 1000 flaky tests but fixing 20 bugs would clear up 900 of those flaky tests.

No idea if it's true. My experience though is that programmer A who has no domain knowledge of the issue marks test as flaky and files a bug. That bug is completely ignored and there's a giant pile of flaky test and people wondering why stuff broke and it turns out because 1000s of tests are being ignored as flaky

We had exactly this at an old job. A flaky browser test that turned out to be due to some misuse of thread local storage, so occasionally users would see data for another user.

Once we tracked that down, we took flaky tests much more seriously.

When thread-local and request-local aren't the same thing...

This is why your CI system should record statistics on which tests failed and when.

A test with a pattern of failing intermittently over its entire lifetime is most likely a flaky test. Disabling the test modifies a bunch of metrics, like skipped tests and code coverage (woe be unto you if some idiot set it up so you can't commit code that lowers test coverage, instead of just warning you).

There are much bigger and more likely sources of regression in the code than a new concurrency bug in code covered by a test that already has a concurrency bug in it. Like someone modifying a test to match the regression they just created.

If there is a concurrency or randomness bug in the untested code, that will probably surface when trying to write a robust test to replace the old one. If not by the author, then by the person the author asks for help when they can't figure it out.

> A test with a pattern of failing intermittently over its entire lifetime is most likely a flaky test.

I'm not GP, but I think they meant something more radical.

In CI, if a test is seen to fail and then succeed on the same git commit for the first time, that's it. One time is enough to classify it as "flaky". No more lifetime, no more patterns of interesting behavior. The test is removed and thrown into backlog.

I might paint with a broader brush. If your commit breaks master, we revert the whole change and you try again.

When the old stuff breaks, there's a bunch of other decisions built on top of it and there is no clear path to unwind the changes. Between HEAD and HEAD-4? Yank it.

If you have a problem with testing infrastructure/code, that will start disabling random tests until either you're left with nothing, or realise that radical moves are rarely practical. (With a side of distrust from any team you send the work to)

Well, if it's a third-party software, that's not ideal but "ok" for realistic levels of ok.

But yeah, your tests shouldn't be non-deterministic unless there's a very good reason why not (and to be honest I can't think of any)

For added resiliency, try changing the order of the tests and/or not running some tests sometimes

That being said, my obligatory test rant is "don't write micro-tests" (like "test if method returns" then "test if method returns the right value" etc) and don't set-up everything in the setup function (that is, setup only what you need).

> Before disabling the test, does anyone even consider the possibility that maybe it's not the test that's flaky, but that the software under test is itself buggy in a non-deterministic way?

I work on the team that build the system in the article. When we disable tests we ask our developers investigate the root cause. Most of the time it is related to application code, so fixing a intermittently failing test definitely isn't related to the test itself, but is often a result of multiple factors.

The key realisation, I think, is that you've already got that problem. A flaky test is one that's failed to do its job. That's why you can't just disable it, you need to back that up with rework to make sure the underlying problem is addressed.

In talking to people about this problem, a common misconception I hear is that "well it only fails 1% of the time (so why should I care?)"

Developers don't think in percentages. Our experiences are based on how frequently something happens, not the percent of the times it happens. In a CI build, where we are agressively building many times a day, your one bad test is going to fail every week or two.

Now add two dozen more tests that do the same, and now builds are failing 2x a day on average and more frequently during crunch times and every few months the build will fail 4, 5 times in a row. And for some reason this happens right when someone actually needs a build now because they broke something and everyone is waiting for them to fix it. Talk about stressful.

Every day is "all the time" to most people, and 5 builds in a row might be "hours" (because you have to keep babysitting it). These are serious broken windows and that is how you should treat them.

The worst thing is you end up normalizing failure - nobody even bothers checking whether it's a "real" failure until it's failed a few times in a row. At that point you might have introduced other test failures which nobody noticed, or someone might have built on top of the broken code, making a fix harder, and so on and so on.

Indeed. This pattern though is often the point at which an intervention is possible. Almost like it has to happen repeatedly before you can get consensus to stop it.

I've worked on the project in the article so I can give some perspective on our handling of intermittently failing tests.

Generally, I agree with your sentiment. Intermittently failing test are eroding trust in the codebase. In our case there are practical challenges with the size of our test base. Not everybody has context on every part of the codebase. This makes it really hard for people to decide if a test is intermittently failing and decide to remove it. The other issue is that at the size of our test base, we are introducing more intermittently failing tests than we can eradicate effectively. To combat this problem we build a test-onboarding system that tests test for common issues. Once a test enters the codebase we are disabling the tests if they are failing and then passing for the same commit too many times so that an individual developer doesn't have to make the decision.

I totally agree with you. We have a monorepo with maybe a hundred thousand tests, and we have a nightly build that runs against all of them, and publishes the results in an email to all engineering. 3/5 days we come out clean. 2/5 days we discover and fix a flaky test or broken build target.

If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet.

Edit: looks like our numbers are quite close to Shopify’s. We don’t need anything close to hundreds of docker containers running in parallel. But we’re mostly a python shop, maybe they’re being a ruby shop slows things down here.

Ruby is not really slowing us down, the bigger issue is that we have a lot of integration tests and few isolated unit tests. So we are massively parallelizing our builds so we can get p50 build times of 23 minutes when running the whole test suite. The time mentioned in the article is a little bit closer to the p95 time.

> If there’s an intensity of flakiness that can withstand the gaze of a couple hundred engineers for more than an hour or so, we haven’t found one yet.

That is a good approach and we also used to disable tests that would fail on full master builds. When it comes to fixing intermittently failing tests we are now storing every test run so we can analyze our test suite. This way we are able to find the slowest and flakiest tests in the codebase, disable them, and notify the owners of the code to investigate.

That’s interesting and definitely helps make the numbers make more sense! :)

In your experience, do you find high value in having your tests be 80/20 (rough numbers) integration/unit vs the other way around?

I guess it depends on the programming style, but here it’s the other way around. Some minority of engineers here clamor for end to end tests every once in a while, but we’ve rarely found the difficulty of establishing the test harness to be worth the marginal gains in reliability. We’re also wary of the temptation to write e2e tests to test unit-test-level correctness, because of the implications on test performance, like you describe.

On our side, the closest thing we have to ubiquitous e2e testing is that we spin up and spin down one ephemeral database instance per test module (could be used by many test functions).

How does your zero tolerance policy work when you can't find a reason the rest fails? I.e. you have critical functionality you test, but fail one test run in 1000. You can spend 2 days on the issue and not find a solution.

Sinking more time into it doesn't really help anyone much and won't offset any time saving in a realistic future. On the other hand, if you actually remove the test, you risk messing up a critical transaction.

If you have a critical function failing testing, you need to figure out the root cause, whether it takes 2 days or 2 months. 2 days is nothing - I've spent more than that on UI polish bugs, to not be able to dedicate 5x that time to fixing critical functionality, something is desperately wrong.

Viraptor is right. Your "solution" is far from unversally acceptable. If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective.

I hate the flaky tests just as much as a next dude but seriously, you need to factor in everything - deadlines, features, CI/CD, metrics etc, while still having constant time frame.

So lets not get into such polar opinions, its just not realistic at all. Life is never that ordered.

If you came to me with that attitude, I would fire you immediately.

This is, per definition, critical functionality, and we can't figure out whether it is working or not, and you are saying you don't care.

> If you came to me as developer telling me that you did 2 months debug on test that fails randmoly once in a week and you didn't do much else, I would probably show you the dor as a team lead. We need to put things into perspective.

This is a pretty polar opinion right here - it's not a flaky test that fails once a week, it's unreliable critical functionality. If I were _your_ lead and you told me you fired someone because they spent two months on something, I'd probably fire you for not knowing what your team are doing for two months.

> This is a pretty polar opinion right here - it's not a flaky test that fails once a week, it's unreliable critical functionality.

It may be unreliable critical functionality, or unreliable test. In some cases it's obvious which one it is, in some it isn't. If you know it's the test, it becomes much less beneficial to spend days on it.

I didn't say fire, didn't I, I said "show you the dor", like, get out from my office :P

And seriously, if you fire someone for having a strong opinion, you should remind yourself that even little kids tollerate a far worse behavior.

If it's acceptable for your critical functionality to fail 0.1% of the time in production, then sure, leave it in.

Meh.

Majority of time, in my experience, flaky tests has nothing to do with the software but it is about faulty test infrastructure.

There's difference between a flakey test and test failing due to randomly failing feature. They're not always the same thing. The test failing does not necessarily mean the feature doesn't work.

The biggest thing is anything time based the effort to mock out of fake time is normally way larger than just dealing with a test that flakes from time to time.

That being said I make a concentrated effort to take in time rather then randomly calling now() in the middle of my code but most code bases aren't green fields.

It is my personal opinion that in that first statement you are severely underestimating the time and product quality that is eventually lost because of a flaky test that isn't dealt with quickly.

We just implemented a rule where tests taking > 100ms to execute are considered a failure. You can experiment with the parameters or limits but this sort of rule has paid huge dividends.

The majority of the tests it has disqualified sucked and didn't really "test" anything. They also tended to be the most brittle.

You don't write tests that hit APIs and databases when you have strict time limits and this eliminates a lot of "flaky" tests.

Testing budgets are important by disqualifying a test because it's "slow" is really silly.

1. Tests that hit APIs and databases are important because when you mock out the database and the API you're writing a test that asserts your code integrates correctly with what you think the database or API should be doing.

2. Some systems are highly asynchronous and thus very difficult to test individual cases in under 100 ms.

If I have a suite of 100 tests that completes in 10 seconds because everything runs concurrently, that's a lot more important than limiting and individual test to some arbitrary timespan.

Again, make a budget for tests and meet it instead of making hard rules about how fast or slow a given test should be.

obviously the context matters, but putting time limits on some tests can really cause headaches.

We've had problems where builds failed when many jobs ran together because build runner contention meant that jobs took just a bit longer than normal.. and of course the re-run would work because contention is lower.. the function still behaves as expected just a bit slower, is that worth failing a test and hurting productivity?

The obvious exception is benchmark performance critical code, but run of the mill functional tests should maybe have an unreasonably high time limit bc it hurts less?

Maybe an area you want to carve out an exception for are benchmarking tests; i.e. tests that ensure that a function runs faster/slower than a certain speed. In this case, you probably need more than 100ms of testing for accurate results.

For example, password checking functions should be slow, and extremely hot/performance sensitive functions should be fast.

Of course, you should have fairly wide margins to avoid flakes.

Are there exceptions? Our codebase has some functions related to generating secure cryptographic keys and certificate chains, and that test suit takes many seconds to run since it actually stresses the CPU.

If you can't hit your database in CI within 100ms or talking to it makes your tests flaky, you got problems.

I assume the parent commenter meant unit tests must take < 100ms. Unit tests should be mocking databases in-memory. So latency to databases in unit tests should be less than a microsecond.

It's okay (and expected) if integration and end-to-end tests take more than 100ms.

My point isn't that it's OK to take more than 100ms for and end to end test. My point is there is no reason why you shouldn't be able to reliably run tests that hit databases, multiple times, within a 100ms deadline and without adding flakiness.

It's 2020, computers are fast. You can do tens of thousands of database transactions in a second. Starting up postgres takes a fraction of a second and consumes negligible resources. If you treat your DB files as a build artifact that includes schema migration.

Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test. The purpose of an end-to-end test is to test the whole system end-to-end; this includes network, other servers, etc.

Each of these can break, which is why e2e tests are inherently more flaky than unit tests. And network latency and transfer speeds mean that taking more than 100ms may happen often.

Another example of an e2e test that takes a long time are screenshot diffs. Spinning up Selenium and rendering a webpage might take more than 100ms in rare instances. iOS/Android tests are in the same vein where reloading an app takes > 100ms much of the time.

> Spinning up a local instance of postgres defeats the whole purpose of an end-to-end test.

So run some end to end test in a fully featured staging-style environment as well? I'm not disagreeing at all that having some environment that closely resembles production for very late phases as well is nice to have. I'm arguing against nonsense like mocking out database calls because of weird superstitions.

> And network latency and transfer speeds mean that taking more than 100ms may happen often.

The operative word being "may". If your staging server in Timbutku needs to talk to your other staging server on the South Pole you will blow your 100ms window all the time. If it all runs within a single good DC, network latency ought to be ~0.1ms roundtrip and you can shovel around Gigabytes per second.

Tests that hit dynamic data stores like DBs don't fail very often due to network speed constraints compared to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems. It's easier to just prevent these situations.

In the abstract I'm not sure what a test that talks to the database is actually testing.

How do you effectively develop if you can't reliably talk to the DB in dev due to configuration, system updates, security, schema changes, and a myriad of other time-consuming hard to diagnose problems?

> In the abstract I'm not sure what a test that talks to the database is actually testing.

Well, presumably the reason that you have a DB in the first place is that your application requires a DB for some of its functionality. So how do you make sure this functionality works if you don't talk to the DB?

Don't get me wrong -- I love that you refuse to accept ridiculously slow tests and enforce a 100ms deadline; I wish everything I was working on had the same approach. But unless you have some specific requirements (like talking to a proprietary cloud DB), you should maybe investigate if you might not have to accept slow and flaky DBs in dev or CI either. My experience is you don't.

Demote it to a group of flaky tests that don't block pushes. Or, demote them and block the push when a flaky test fails 100% of the time.

Then what’s the point of them?

That gives me an idea: what if you had a pool of flaky/failing tests, but it was limited in size, and if you add a test to that pool that takes it over the maximum size, you must either fix it, or evict another flaky rest by fixing it.

You could use the pool capacity and size as a proxy for technical debt. Starting a new, large project? You can increase the pool size temporarily until the project reaches mvp. Periodically the size of the test pool should be evaluated, tests fixed and the pool size shrunk.

Over time the overall pool size should shrink or at least stabilise.

I imagine without any discipline though you just get a consistent set of tests that no-one has the bandwidth to go and fix or make unflaky. I wonder if flakiness of tests correlates with discipline to paying down tech debt and improving testing

They still detect real failures. You want to know when they go from failing 10% of the time to 100%. If your team wants, block pushes at 100% failure.

But how is it a reliable indicator of any sort of failure if it doesn’t always fail? Doesn’t it erode trust?