There are tons of systems that setup environments on-demand (EaaS) to facilitate "pre-merge" tests. Im sure your problem can be solved without building it all yourself.

Also noteworthy: Slack is going to explain this in more depth next month:

https://www.linkedin.com/video/event/urn:li:ugcPost:69626375...

Or you can read the article: https://slack.engineering/remote-development-at-slack/

This is possible with github actions. Have one job create a matrix of other jobs for each of your runners.

Use self hosted runners if you are worried about costs or have special requirements.

Test build is dozens of gigabytes and millions of files and takes 20+ minutes.

When setting up github actions matrix, we have a choice of either (1) build before every test (+20 minutes per test) or (2) making a huge docker image and upload it, and download it for each worker, which also is 20+ minutes or more.

You can use github actions if you just use github to allocate nodes, and do all the work with your custom test scheduler, but then you are starting on the road of rewriting the CI/CD for yourself.

There are limits on build queue sizes unless you buy the enterprise plan. Also doesn't work if you are not hosting code on GitHub

Did you consider leveraging buildbot (buildbot.net) when writing your infrastructure? It has some features that should be helpful.

I'm currently setting up a ci/cd build infrastructure for myself. While I've been exploring different ways to perform the build/test cycle, I expect it will be beneficial to wrap the whole thing in buildbot in the end.

I would think running something within a Docker container is running it natively?

Test build is dozens of gigabytes and millions of files (yeah, we have some suboptimality in how we set up bazel...), so even simple "rm -rf" can take 30+ minutes. Docker really has too high overhead for this, especially in the cleanup stage.

How do you build a container from inside a container? Answering this gives you one example of a difference