It's not just true for GHA, but for any build server really: The build server should be a script runner that adds history, artifact management, and permissions/auditing, but should delegate the actual build process to the repository it's building.
At the same time, I've found that it often isn't sufficient to push everything into a proper programming language: I do sometimes (even frequently) need to use vendor-specific functionality in GHA, mark dependencies between jobs, invoke REST APIs that are already well abstracted as actions, etc. Re-implementing those things in a programming language of my choice is possible, but doesn't break the vendor dependency and is (IME) still brittle.
Essentially: the vendor lock-in value proposition for GHA is very, very strong. Convincing people that they should take option (2) means making a stronger value proposition, which is pretty hard!
Another thing I noticed is that if you do 1 sophisticated features like build caching and parallelization often becomes completely impractical whereas if you default to 2 you can probably do it with only a moderate amount of commit-push-debug.
A CI "special" language is almost by definition something that can't be run locally, which is really inconvenient for debugging.
Ideally you want your scripting to handle of the weird gotchas of different versions of host OSes, etc. Granted my work is cross-platform so it is compounded.
So far I’ve found relying on extensive custom tooling has allowed me to handle transitions from local, to Travis, to AppVeyor, to CircleCI and now also GitHub Actions.
You really want your CI config to specify the host platform and possibly set some env vars. Then it should invoke a single CI wrapper script. Ideally this can also be run locally.
FWIW, there's nektos/act[^1], which aims to duplicate GHA behavior locally, but I haven't tried it yet.
Can't you just use the Github API for that? The script would be triggered by the YAML, but all logic is inside the script.
But `act` is cool, I've used it for local debugging. Thing is its output is impossibly verbose, and they don't aim to support everything an action does (which is fine if you stick to (2)).
The main thing that you can't decouple from GHA is pushing and pulling intermediate artifacts, which for some build pipelines is going to be a pretty big chunk of the logic.
> can I push this YAML complexity into a script?
- what language is the script written in?
- will developers use the same language for all those scripts?
- does it need dependencies?
- where are we going to host scripts used by multiple github actions?
- if we ended up putting those scripts in repositories, how do we update the actions once we release new version of the scripts?
- how do you track those versions?
- how much does it cost to write a separate script and maintain it versus locking us in with an external github action?
These are just the first questions that pop in my mind, but there is more. And some answers may not be that difficult, yet is still something to think about.
And I agree with the core idea (move logic outside pipeline configuration), but I can understand the tepid reaction you may get. Is not free and you compromise on some things
#!/usr/bin/env bash
set -ex
aws send-email ...I think the point about maintenance cost is valid, but the thesis of the comment that you are responding to is that the prebuilt actions are a complexity trap.
I think of it like this:
I write scripts (as portable as possible) to be able to build/test/sign/deploy/etc They should work locally always.
GitHub is for automating me setting up the environments where I can run those scripts and then actually running them.
I always opt for option 2 where possible though.
May have misread this but you know you can push to one branch and then run the action against it? Would reduce PRs if you're doing that to then check the action in master. You have to add a workflow_dispatch to the action: https://docs.github.com/en/actions/using-workflows/manually-...
git commit --amend --no-edit && git push -fThe devil's in the details, etc, etc, but I think it's a _much_ more sane default, even for single-user setups/branches because accidents can happen and git DGAF
Most of the CI systems have some type of plug-in for Docker builds. Drone has a Docker plugin, but I don't use it. If you look at all of them, you'll notice that you end up with a single build step that does everything Docker related; build, tag, push.
What if you want to add something between build and push? It's difficult (or impossible) and you start to end up with one gigantic, do-everything build step. Just having a flag and a conditional in the build step to enable/disable pushing bugged me because 'push' is a distinct command with Docker, so I think it makes more sense if 'push' is a distinct build step. Nothing supports that though and you seem to end up with the same "do-everything" build step on every platform.. The vendor lock-in caused by #1 doesn't even matter at that point because your build logic is married to the the underlying Docker build plugin of the CI system.
Instead, I went for something that does a better job of mirroring what you get on the CLI when you run Docker by hand. I wanted distinct build steps for build, test, tag, push. That's what I do on my dev machine, so, in my opinion, that's what should be happening in CI. That way I can debug by manually running each step locally, just like I would without a CI system.
What I do with Drone is ignore their plugin and start a detached process to provide a Docker daemon. The default is to start a Docker-in-Docker daemon, but, if I'm running locally, I can pass --instance 'docker.localhost' and it starts a Caddy server that proxies to the local Docker socket instead of running a Docker-in-Docker container.
Every build step after that can access the Docker daemon. It's scoped to the pipeline.
My 'build' step is pretty much 'docker compose build --pull', so I can do the same on the CLI to debug my Docker build without touching the CI system. If I want to use the CI system I can add that '--instance' flag to bind to the local daemon which is quick and takes advantage of caching. I can also do a default build locally which spins up a Docker-in-Docker container. That mimics what the CI agent will be doing. Finally I can push commits and let it run on the CI system.
Every build step after the setup uses the 'docker.io/docker' image, so the small number of scripts it takes to get everything set up are executing in a fairly well known environment.
I purposely avoid caching by default. I think it adds too much complexity. I use Nexus as a local cache for all artifacts (Docker, APT, etc.) which is an ok compromise and ensures I'm not hitting the upstream services more than needed.
If someone took Drone away from me tomorrow I could do my builds from the CLI without even thinking about it. I think it's a good habit to get into to ask yourself what would happen if you lose a tool before getting too committed to it.
I recommend making these scripts cross-platform for flexibility. Use matrix: and env: to handle it. Go for Perl, JavaScript, or Python over OS shells and put file tasks in scripts to dodge path issues.
I've tried boxing these scripts into steps, but unless they're super generic for everyone, it doesn't seem worth it.
https://lobste.rs/s/veoan6/github_actions_could_be_so_much_b...
Also, seamlessly interleaving shell and declarative JSON-like data -- without YAML -- is a main point of http://www.oilshell.org, and Hay
Hay Ain't YAML - https://www.oilshell.org/release/0.18.0/doc/hay.html
I won’t go into the details on why it’s this way (build chain madness). It’s stupid and necessary.
Generally, I would use the same tools (e.g. ./gradlew build or docker build) to build stuff locally as on CI, and config params are typically enough to distinguish what needs to be different.
My CI scripts still tend up to be more complicated than I'd like to (due to things like caching, artifacts, code insights, triggers, etc.), but the main build logic at least is extracted.
Maybe this means I should encapsulate this into an action, and check it in somewhere else. But I don't really feel like that; an action is a lot of overhead for a 20-line bash script. Not to mention that erases the lack of lock-in that the script alone gives me.
I guess I could check the script into a separate utility repo, and pull it into my other repos via git submodules? That's probably the least-bad solution. I'd still have to update the submodule refs when I make changes, but that's better than copy-pasting the scripts everywhere.
As in, use GitHub Actions as a YAML wrapper around bash/zsh/sh scripts?
Python is actually preinstalled on Github CI runners.
Is it possible to write Python package that based on YAML specification produces Python API? User will code in Python and YAML will be the output.
I was working on YAML syntax for creating UI. I converted it to Python API and Im happy. For exmple, dynamic widgets in YAML were hard, in Python they are strightforward.
I think you explained quite well what I couldn't put my finger on last time: Building every simple workflow out of a pile of 3rd party apps creates a lot of unnecessary complexity.
Since then, I have used GitHub actions for a few projects, but mostly stayed away from re-using and combining actions (except for the obvious use cases of "check out this branch").
Compared to Gitlab CI, GH Actions still feels like a toy unfortuantely.
They don't seem to grasp how bad their setup is, and consequently are willing to understand awful programming conditions. Even punch cards were better as these people had the advantage of working with a real programming language with defined behaviour. "when exactly is this string interpolation step executed? in the anchor or when referenced? (well, it depends)". No it's black box tinkering (you might as well be prompt engineering)
the C in IaC is supposed to stand for code. Well, if you're supposed to code something you need to
- be able to assert correctness before you commit,
- be able to step through the code
(sorry for the rant)
Adding state? Adding conditionals? Adding (more than a couple) procedure calls?
These concepts perform poorly without common programming tools: testing (via compilation or development runtime), static analysis, intellisense, etc etc
Imagine the curve:
X axis is (vaguely) LinesOfYaml (lines of dsl, really) Y axis is tool selection. Positive region of axis is “use a DSL”, lower region is “use a GeneralPurposeProgrammingLanguage”
The line starts at the origin, has a SMALL positive bump, than plummets downwards near vertically.
Gets it right? Tools like ocurrent (contrasted against GH actions) [1], cdk (contrasted against TF yaml) [2]
Gets it wrong? Well, see parent post. This made me so crazy at work (where seemingly everyone has been drinking the yaml dsl koolaide) that i built a local product simulator and yaml generator for their systems because “coding” against the product was so untenable.
[1] https://github.com/ocurrent/ocurrent/blob/master/doc/example... [2] https://docs.aws.amazon.com/cdk/v2/guide/getting_started.htm...
Independent tutorials unfortunately fall in the same bucket as they first look at official documentation to try to follow so-called best practices or just try to get their things working, and I would say also because shell scripts will seem more hacky for many people -unfairly-.
1) Program with github actions. Google "how can I send an email with github actions?" and then plug in some marketplace tool to do it. Your workflows grow to 500-1000 lines and start having all sorts of nonsense like conditionals and the YAML becomes disgusting and hard to understand. Github actions becomes a nightmare and you've invited vendor lock in.
2) Configure with github actions. Always ask yourself "can I push this YAML complexity into a script?" and do it if you can. Send an email? Yes, that can go in a script. Your workflow ends up being about 50-60 lines as a result and very rarely needs to be changed once you've set up. Github actions is suddenly fine and you rarely have to do that stupid push-debug-commit loop.
Every time I join a new team I tell them that 1 is the way to madness and 2 is the sensible approach and they always tepidly agree with me and yet about half of the time they still do 1.
The thing is, the lack of debugging tools provided by Microsoft is also really not much of a problem if you do 2, vendor lock in is lower if you do 2, debugging is easier if you do 2 but still nobody does 2.