We built the fastest CI in the world and it failed

Re:We built the fastest CI in the world and it failed by danudey on Tuesday September 12, 2023 @12:12PM (#37483479)

That's step one, 'how to write a basic Earthfile that just does the thing', so they don't delve into the details of caching and how it works, but look at the whole block and not just that one line; it operates like a Dockerfile:

    build:
        COPY build.gradle ./
        COPY src src
        RUN gradle build
        RUN gradle install
        SAVE ARTIFACT build/install/java-example/bin /bin
        SAVE ARTIFACT build/install/java-example/lib /lib

The COPY phases copy files into the build context. If the files haven't changed, then the commands don't need to be run. In other words, if `build.gradle` and `src` are unchanged, it won't run `gradle build` or `gradle install`, and it'll just give you the artifacts from the previous build.

They have a golang example in part three[0], which they redesign to use caching effectively:

    build:
        # Download deps before copying code.
        COPY go.mod go.sum .
        RUN go mod download
        # Copy and build code.
        COPY main.go .
        RUN go build -o output/example main.go
        SAVE ARTIFACT output/example AS LOCAL local-output/go-example

They copy in go.mod and go.sum and then run `go mod download`. If the mod and sum files haven't changed, then `go mod download` doesn't need to be run.

Then they copy `main.go` in, and run `go build`. If the `main.go` file hasn't changed, `go build` doesn't need to be run.

[0] https://docs.earthly.dev/basics/part-3-adding-dependencies-w...

[ Parent | on HN ]

Re:We built the fastest CI in the world and it failed by brabel on Tuesday September 12, 2023 @12:51PM (#37484099)

Gradle can do the caching of tasks itself in a level of granularity no CI can. If you keep your build cache "live" in the CI, Gradle knows to only recompile what has changed, and to only run tests which changed (or tests something which changed). I am not sure if many other build systems do that, but this is something I would absolutely expect from the build system, not CI. Do you not expect caching when building locally??

The whole idea with CI doing this is wrong IMO. All I want CI to do is call a couple of commands after setting up the environment (which would include the Gradle cache or whatever to speed up things - no need for your revolutionary CI build system).

[ Parent | on HN ]

Re:We built the fastest CI in the world and it failed by nvm0n2 on Tuesday September 12, 2023 @02:01PM (#37485014)

A CI system like TeamCity will give you that sort of thing for free though, without needing an Earthfile. It just dedupes builds at the same commit, and keeps checkout directories between builds on the agents, so if the build tool has a cache then it's automatically reused. Works great.

Given that it's hard to understand what exactly this is for? Build systems normally try to avoid redundant work for local purposes anyway. Is it some hack for companies that insist on resetting build environments to zero on every single build, and then decide they want caching back? If so, why not just ... let the builds take place in directories containing previous builds, and fix the problems? That's what we do at my company and it works great (using gradle), there are only rarely problems with unclean builds and it's easy to rerun with a clean checkout if that's suspected to be an issue (the bugs are mostly in our own code which does a layer of caching above the build system anyway).

[ Parent | on HN ]

Re:We built the fastest CI in the world and it failed by oftenwrong on Tuesday September 12, 2023 @09:41PM (#37491157)

[delayed]

[ Parent | on HN ]

Re:We built the fastest CI in the world and it failed by nvm0n2 on Wednesday September 13, 2023 @04:05AM (#37493625)

So they claim. And yet ultimately they do it by caching things, same as every other system.

[ Parent | on HN ]

Re:We built the fastest CI in the world and it failed by theamk on Tuesday September 12, 2023 @03:14PM (#37486347)

sure, but in a java project, how likely you are to have a commit with no changes to src/? or for a go project, how likely is it that you want to duplicate you entire dependency tree in a clunky, non-parallel dependency description language, vs just doing "COPY * .; RUN go build" and letting golang figure it all out and build in parallel using all the cores?

Don't get me wrong, there is definitely value in caching dependencies, and I can see Earthly approach help if you have, say, a Python app, with 10 minutes of dependency install time, 0 build time, and 10 seconds of tests - you'll see huge improvement. But you'll also see almost the same improvement if you use Dockerfile on a persistent host instead, and as a bonus you won't have to learn new system.

One place Earthly might shine are monorepos with multiple independent projects that even use multiple build systems... but I am not sure how many of those exist _and_ don't have some other caching solution already.

[ Parent | on HN ]

Re:We built the fastest CI in the world and it failed by yencabulator on Wednesday September 13, 2023 @11:47AM (#37498296)

That only splits dependency-build from app-build. Any change to the app source code still causes a full rebuild, unless you figure out how to share the cache directory between builds (and then parallelism gets sketchy).

Generic CI just simply can't parallelize+cache on a level comparable to systems that understand the build itself (Bazel, sccache, etc).

[ Parent | on HN ]