Logging in via SSH seems useful although that would mean keeping the machine running after a failed build.
For example, Gitlab CI has a local runner but it's not really first class, definitely isn't kept up to date with the real runners. It seems like it's more of a recreation of a job system based off of (an older version of) Gitlab CI configuration files.
Circle CI has a more decent story around this if I remember correctly. But first-class support for local reproduction of CI runs is the kind of thing I'd like to see as table stakes today.
It can be challenging to debug certain things in the live CI system for developers without being able to recreate locally (otherwise you have to bring in the team that manages the CI system to help them debug with you which takes time).
I don't understand why this would be true.
For $dayjob I've been writing end-to-end tests that run via GitHub Actions. Each test starts with a package of the compiled code (built by another GHA workflow), deploys a bunch of cloud resources including a VM, scp's the package to the VM, and ssh's to the VM to install the package and run the test. The package contains multiple services that must be individually configured based on the particular test, and then a test process interacts with these services in a bunch of ways. I think we can agree these are complicated tests; they're not just unit tests you can run in a Docker container and be done with (which we also have, of course).
Now, these tests are written entirely in a shell script. It has been designed from the start that a developer can run the script locally and get exactly the same result as when GHA runs it. The things like auth tokens for the cloud are read as environment variables, so when running as a GHA workflow the env vars are hooked up to repository secrets, but when running locally the developer can just provide their own auth token via env vars to the script. Furthermore, the script cleans up the cloud resources when it finishes, but if the developer wants to investigate the failure they can just comment out the cleanup step from the script.
I'm sure there are third-party GH Actions to create those cloud resources, to scp files to remote targets, and to run commands via ssh against remote targets. But using those actions not only removes the ability to run the tests locally, it also requires learning arbitrary DSLs ("How does this action take parameters? What limitations does it have? How do I process the output of one action to be the parameter of the next? How do I parallelize these two actions that don't need to run in series?") and introduces third-party code that you have to trust with your cloud auth token.
The problem is that we need a ton of logic in the coordinator. For example on how to do matrix builds. For local builds to be perfectly similar we would have to recreate that logic inside the runner. This is hard, not in the least because it are different programming languages.
The one that made me go "oooh!" is this:
Submitting jobs without pushing and logging in after failed builds are two features that I didn't know I wanted. But now that I see them, I really want them.