News for hackers. Stuff that matters.
Re:Sourcehut's Second Year in Alpha by Macha on Sunday November 15, 2020 @01:24PM (#25103400)
The trick is to do as much as possible outside the CI system. Ideally, each step should just call a single script, Makefile command, or your language's build tool equivalent.
Re:Sourcehut's Second Year in Alpha by zbentley on Sunday November 15, 2020 @02:45PM (#25104109)
That gets deeply complicated where integration or automation tests are concerned, and those types of tests provide the most value per effort in environments where test-authoring, QA, and refactoring time is limited.
Re:Sourcehut's Second Year in Alpha by Arnavion on Sunday November 15, 2020 @03:00PM (#25104253)
>That gets deeply complicated where integration or automation tests are concerned

I don't understand why this would be true.

For $dayjob I've been writing end-to-end tests that run via GitHub Actions. Each test starts with a package of the compiled code (built by another GHA workflow), deploys a bunch of cloud resources including a VM, scp's the package to the VM, and ssh's to the VM to install the package and run the test. The package contains multiple services that must be individually configured based on the particular test, and then a test process interacts with these services in a bunch of ways. I think we can agree these are complicated tests; they're not just unit tests you can run in a Docker container and be done with (which we also have, of course).

Now, these tests are written entirely in a shell script. It has been designed from the start that a developer can run the script locally and get exactly the same result as when GHA runs it. The things like auth tokens for the cloud are read as environment variables, so when running as a GHA workflow the env vars are hooked up to repository secrets, but when running locally the developer can just provide their own auth token via env vars to the script. Furthermore, the script cleans up the cloud resources when it finishes, but if the developer wants to investigate the failure they can just comment out the cleanup step from the script.

I'm sure there are third-party GH Actions to create those cloud resources, to scp files to remote targets, and to run commands via ssh against remote targets. But using those actions not only removes the ability to run the tests locally, it also requires learning arbitrary DSLs ("How does this action take parameters? What limitations does it have? How do I process the output of one action to be the parameter of the next? How do I parallelize these two actions that don't need to run in series?") and introduces third-party code that you have to trust with your cloud auth token.