Running dotnet test on the other hand with 10k empty tests took 95 seconds.
Honestly, 10k no-op tests should be limited by disk IO, and in an ideal world would be 10 seconds.
How many tests did you put in each file?
If the unit tests are slow enough to want to skip them, they likely are not unit tests but some kind of service-level tests or tests that are hitting external APIs or some other source of a bad smell. If the slow thing is the build, then cache the artifact keyed off the directory contents so the step is fast if code is unchanged. If the unit tests only run for a package when the code changes, there is a lack of e2e/integration testing. So, what is OP's testing strategy? Caching? It seems like following good testing practices would make this problem disappear.
For example, tests involving read hardware can only run at 1x speed, so you will want to avoid running those if you can. If you are building a custom compiler toolchain, that is slow and you will want to skip it if the changes cannot possibly affect the toolchain.
> In GitHub you can specify a "required check", the name of the step in your pipeline that always has to be green before a pull request is merged. As an example, I can say that web-app1 - Unit tests are required to pass. The problem is that this step will only run when I change something in the web-app1 folder. So if my pull request only made changes in api1 I will never be able to merge my pull request!
Continuous Integration is not continuous integration if we don’t test that a change has no deleterious side effects on the rest of the system. That’s what integration is. So if you aren’t running all of the tests because they’re slow, then you’re engaging in false economy. Make your tests run faster. Modern hardware with reasonable test runners should be able to whack out 10k unit tests in under a minute. The time to run the tests goes up by a factor of ~7-10 depending on framework as you climb each step in the testing pyramid. And while it takes more tests to cover the same ground, with a little care you can still almost halve the run time replacing one test with a handful of tests that check the same requirement one layer down, or about 70% moving down two layers.
One thing that’s been missing from most of the recent CI pipelines I’ve used is being able to see that a build is going to fail before the tests finish. The earlier the reporting of the failure the better the ergonomics for the person who triggered the build. That’s why the testing pyramid even exists.