That gives me an idea: what if you had a pool of flaky/failing tests, but it was limited in size, and if you add a test to that pool that takes it over the maximum size, you must either fix it, or evict another flaky rest by fixing it.

You could use the pool capacity and size as a proxy for technical debt. Starting a new, large project? You can increase the pool size temporarily until the project reaches mvp. Periodically the size of the test pool should be evaluated, tests fixed and the pool size shrunk.

Over time the overall pool size should shrink or at least stabilise.

I imagine without any discipline though you just get a consistent set of tests that no-one has the bandwidth to go and fix or make unflaky. I wonder if flakiness of tests correlates with discipline to paying down tech debt and improving testing

They still detect real failures. You want to know when they go from failing 10% of the time to 100%. If your team wants, block pushes at 100% failure.

But how is it a reliable indicator of any sort of failure if it doesn’t always fail? Doesn’t it erode trust?