With actions you can run multiple tasks in parallel, restart failed portions without retrying the whole CI, run certain sections with conditions like which branch you are on (different tasks when commit to master vs feature branch)

>With actions you can run multiple tasks in parallel,

That is the "strategy" part of the workflow, not the "tasks" part. My comment was about the latter. ("and make your GH Actions run `make` in a script task.")

>restart failed portions without retrying the whole CI, run certain sections with conditions like which branch you are on (different tasks when commit to master vs feature branch)

So can a shell script.

> Running things in parallel is literally a single `&` character.

Yes now try waiting for all parallel tasks, and error out if at least one errors. And try separating their output so that they don't get interleaved into a big mess. And try by default hiding the outputs of commands that succeeded, only showing those that failed, except when the user explicitly asks for it.

Your "simple" shell script now suddenly isn't so simple anymore.

That’s what GNU Parallel is for. Or Pueue, which gives you a GH-Actions-level feature set but it’s less likely to be installed on any particular machine. Pretty sure you could fetch a pueue binary at the start of an actions script and do everything that way.

Using GNU make you can get very nice output sync:

    all: variant-1 variant-2
        @echo done with all
    
    variant-%:
        @echo starting $@
        @sleep 1
        @echo done $@

Use `make -j2 --output-sync=target`.

    brew install make
    gmake -j2 --output-sync=target

on macOS.

>Yes now try waiting for all parallel tasks,

`wait`

>and error out if at least one errors.

`wait` propagates the exit status of the waited task. `set -e` triggers the script to exit on any command failure.

>And try separating their output so that they don't get interleaved into a big mess.

`| while read -r line; do echo "task X: $line"; done`

Learn the tools instead of being so confident that it can't be done or that it's complicated.

You remind me of that guy who said there's no point in Dropbox because one can "just simply setup an sftp server".

What I mean is if you have 50,000 unit tests, writing an actual CI config will let you split that up in to 20 jobs that all run at once and if a test fails, you can retry that 1/20th of the tests instead of the whole thing.

The CI runners aren’t multi core VMs I believe so you can’t just use standard shell utilities, you have to indicate to the CI system you want to run multiple tasks.

Yes, that's the "strategy" part.

I agree. GitHub Actions should call your scripts but your scripts should not depend on GitHub Actions API.

I also suggest Bazel as a consideration alongside Make. With Bazel, you get two advantages over Make:

1. It is easier to ensure that what GitHub Actions runs and builds is the same as what you have locally, since Bazel can fetch toolchains as part of the build process

2. Using a Bazel remote cache, you do not have to repeat work if the build fails halfway and you need to make some changes before running it again.

Same can be said about Nix and as a bonus you get Nix instead of Starlark

Nix & Make is the True Engineer(tm)'s Bazel.

Are Nix and Bazel mutually exclusive?

https://www.tweag.io/blog/2018-03-15-bazel-nix/

This is a lesson that I've learned after going all-out on actions once.

Now my makefiles in addition to the usual "make" and "make test" also support "make prerequisites" to prepare a build environment by installing everything necessary and "make ci" to run everything that CI should check. With actual implementation being scripts placed under "scripts/ci".

The scripts do provide some goodies when they are run by GitHub Actions – like folding the build logs or installing dependencies differently – but these targets also work on the developer machine.

What about caching to reduce ci time? GH setup scripts cache dependencies in a way that would seem hard to replicate in a make file.

If it’s manageable – just don’t. Build from scratch. Make sure your build works from scratch and completes in acceptable timeframe. If it’s painful, treat the root cause and not the symptoms.

If it’s unbearable due to circumstances out of your control, there’s nothing wrong with adding some actions/cache steps to .github/workflows – this goes around the build: fetch previous cache before, update the cache after if needed.

The build is still reproducible outside of GitHub Actions, but a pinch of magic salt makes it go faster sometimes without being an essential part of the build pipeline married to GitHub.

If you need to install a whole host of mostly static dependencies, GitHub Actions support running steps in arbitrary Docker container. Prepare an image beforehand, it can be cached too, now you have a predictable environment. (The only downside is that it doesn’t work on macOS and Windows.)

> If you need to install a whole host of mostly static dependencies, GitHub Actions support running steps in arbitrary Docker container. Prepare an image beforehand, it can be cached too, now you have a predictable environment. (The only downside is that it doesn’t work on macOS and Windows.)

Actually I use a similar workflow for some of the other projects that I have to work with, keeping everything CI agnostic. I incrementally add various types of dependencies to the container images where the application will be built.

For example:

  1. common base image (e.g. Debian, Ubuntu, or something like Alpine, or maybe RPM based ones)
  2. previous + common tools (optional, if you want to include curl or other tools in the container for debugging or testing stuff quickly)
  3. previous + project runtime (depending on tech stack, for example OpenJDK for Java projects)
  4. previous + development tools (depending on the tech stack, typically for pulling in dependencies, like Maven, or npm, or whatever)
  5. previous + project dependencies (if the project is large and the dependencies change rarely, you can install them once here and the changing 5% or so later)
  6. previous + project build (including things like running tests, typically multi-stage with the build and tests, and built app handled separately)

Compared to the more "common" way to do things, step #5 probably jumps out the most here, I do a pass of installing all of the dependencies, say, every morning, or hourly in the background, so that later when the project is built the CI can just go: "Hmm, it seems like 95% of the things I need here are already present, I'll just pull the remaining packages (if any)." Clean installs only need to be done when packages are removed, which is also reasonably easy to do.

Though the benefits of this aren't quite as staggering, if you use a self-hosted package repository like Sonatype Nexus, which can cache any dependencies that you've used previously and make everything faster on the network I/O side. This only doesn't hold true when actually installing the packages takes up the majority of the time (e.g. compiling native code), in which case the above is still very useful.

So, an example of how the stages might look, is as follows:

  Builder: Ubuntu + tools (optional) + OpenJDK + Maven + project dependencies + project build (and run tests)
  Runner:  Ubuntu + tools (optional) + OpenJDK + built project from last image (using COPY with --from, typically .jar file or app directory)

Of course, things are less comfortable when you don't have all of your app's dependencies packaged statically but need them "on the system" instead, like Python packages or Ruby Gems, but then your builder and runner will simply look more alike.

For my own personal stuff I also use a slightly simplified version of this, about which I wrote on my blog here, the drawbacks included: https://blog.kronis.dev/articles/using-ubuntu-as-the-base-fo...

You can split your make file in two CI steps, one cached, and the other one depending on it.

om.

GH setup scripts cache dependencies in a way that is hard to replicate --- full stop.

This so hard. I like to think of the make targets, e.g. build, test, install, etc. as an API that should be consistent across repos. This really helps with cross team collaboration. The details of how these tasks happen is free to change at will without the need to “distribute” these changes to developers. There’s no disruption to anyone’s flow. Plus, with a little documentation, on boarding new developers is so much more simple.

I use Makefiles anywhere I can fit them they're a brief respite from YAML hell. This issue was solved in 1976 -- I appreciate there's a lot of VC money in reinventing the wheel (and coming full circle) but I digress.

I too use Make everywhere, but what I would give for an improved tool that had better syntax, composability, and simultaneously deployed everywhere. Sadly, it is good enough, so we shall suffer forever.

I’ve run into this with overly complicated Jenkins pipeline files as well. I think the root cause is just that a single entry point pipeline is boring— everyone wants a CI config that sets statuses and posts results and does things in parallel and interacts with plugins, and every one of those steps is something that is at least semi unique to the CI execution environment.

I think the method you describe is still absolutely how it should be, but this types of interactions are why there’s gravity in the other direction.

>every one of those steps is something that is at least semi unique to the CI execution environment.

Apart from triggers and environment set up none of those things have to be unique.

I often push complex CI logic in YAML into code where it is more easily debugged and I dont have to scratch my head to figure out how to use conditionals. Sending slack messages should always be in code IMHO.

This. All my action files are just a `make test` call which installs dependencies on-demand and most importantly, makes the process reproducible locally which is invaluable to debug.

A Makefile really sucks at displaying outputs/logs of commands, especially when there are lots of commands and when they run concurrently. It also really sucks at communicating what the overall progress is: how many jobs have finished, how many left, how much time has elapsed.

Heck make can make all this much better by just prepending each output line with some colored prefix and timestamp. But make hasn't changed in 30 years and likely won't change.

People are proud that it "solves" things since 1976. Yes if your requirements never changed since 1976. I'm not holding by breath that it will deliver basic usability-enhancing features that one can reasonably expect nowadays.

You can just invoke make build, make test within the specific CI stages.

Then you reduce make to a simple small-scale task runner, basically admitting that it's unusable for large numbers of heterogeneous tasks or concurrency.

Can anyone point me out to a GH repo with this or similar setup. Eager to learn, especially if it runs a bit more complex node/next.js build.

me too, examples please!

Any examples, blog posts on how to use Makefiles for CI?

forget make, just put it all in a docker container and let the docker container be your CI. that's what the tool linked in the post does, and if you have a docker container you can run it unmodified in just about any CI system.

This is what I do except I use a shell script instead of a Makefile.

A working example of this is at: https://github.com/nickjj/docker-flask-example/blob/912388f3...

Those ./run ci:XXX commands are in: https://github.com/nickjj/docker-flask-example/blob/912388f3...

I like it because if CI ever happens to be down I can still run that shell script locally.

You should still have a Makefile that calls your shell scripts when "make" or "make test" is run. Every person who writes shell scripts has a different filename and arguments. "make" and "make test" are always the same everywhere.

> installing $language_compiler

Yes, because proper use of the tool cache (and other caches) significantly speed up GitHub Actions builds.

Look, I'm all for putting logic in Makefiles instead of in YAML. But not at the expense of slower (and therefore more expensive) builds!

Strongly concur. I'll concede some cons:

* GNU make is sometimes unavailable

* syntax is an acquired taste

* not everything fits in a rule body

Imho, these are far outweighed by the flexibility, portability, and "least surprise" convention embodied in a Makefile.