I mean if this works for you, great, but it sounds like a dangerous dynamic to aim for.

The occasional freezes mean future tests written do not need to be optimised (there will be a freeze for important releases anyway), so the unoptimised-tests problem will grow worse the longer you use this strategy, and the cost to rectify it will grow, until your release frequency collapses on itself and everyone complains that "there's no solution to this problem".

These are the sorts of things I think it's very important to nip in the bud. Spend the two weeks now, when it only takes two weeks! Much better than putting everything on hold for six months to fix it four years from now when it becomes untenable.

>> The challenge for us is it'd likely take an engineer a week or two to optimize our test suites. That's enough time for them to ship meaningful features.

That does seem like a possibly worthwhile choice though? A "stop chopping the tree to sharpen the axe" moment?

That might make sense if it'll make or break you. If successful you can afford to fix it later.

But, maybe spend a day on it. I've been to places where the build took hours and there were obvious 30min solutions to fix it (which I did, in some cases).

Consider it an investment. 2 weeks now gives you how many more features over the next six months due to increased dev velocity?

It’s usually cheaper to just add another CI runner to run the tests until the CI runners cost more than the dev time to speed it up.

The slowness in my tests comes not from having too few runners (we just dynamically spin up a runner for each commit, and it normally takes only a few seconds for that), but from serialization in the CI pipeline. You cannot run tests until the code has been built; no extra runner can change that.

I need dev time to make the builds understand their inputs, and then cache more aggressively than we are, or something. (But I lack the dev time.)

pipeline time is something that I spend a non trivial amount of effort on.

caching and promoting images is an absolute must. caching images has it's downsides, too though. it can be a longer time period between when something in the dockerfile breaks and when it's noticed.

generally speaking, my pipeline is generally 10-40% build time, 50-60% integration & system test time, 10% deployment time

The system tests that're run deploy the entire dependency graph of services that my project depends on, which can take a bit to settle, has a high chance of transient failures relative to other parts of the pipeline, and notably has a fixed test run time.

That last bit really raises the lower bound for how long the pipeline as a whole takes -- number of runners doesn't matter.

The pipeline generally takes between 25-45 minutes, and runs on each push and is triggered by changes to dependencies to ensure backwards compatibility.

I wouldn't do anything to make it take longer than it does now, and it can be frustrating. but the pipeline will catch things that would otherwise slip through the cracks.

Same thing goes here. It works great until it doesn't.

E.g. we went with that route with GH Actions, until our tests didn't fit into the memory of the provided machine sizes anymore (shouldn't be a problem anymore as GH provides options for bigger runners now). So we had to spend 1-2 days on setting up a self-hosted runner in a way that it wouldn't be too heavy in maintenance, and then probably another 1-2 days of rework over the next few weeks to iron out the kinks.