aayushshah15

Yes! This is documented in our docs: https://docs.blacksmith.sh/blacksmith-caching/docker-builds#..., the TLDR is that you can use the `build-push-action` with `setup-only: true`.

[cofounder of blacksmith here]

This is on our radar! The primitives mentioned in this blog post are fairly general and allow us to support various types of artifact storage and caching protocols.

> one downside I've observed with some serverless in the past is that it can introduce up-front latency delays as the system spins up support to handle your spike

[cofounder of blacksmith here]

This is exactly one of the symptoms of running CI on traditional hyperscalers we're setting out to solve. The fundamental requirement for CI is that each job requires its own fresh VM (which is unlike traditional serverless workloads like lambdas). To provision an EC2 instance for a CI job:

- you're contending against general on-demand production workloads (which have a particular demand curve based on, say, the time of day). This can typically imply high variance in instance provisioning times.

- since AWS/GCP/Azure deploy capacity out as spot instances with a guaranteed pre-emption warning, you're also waiting for the pre-emption windows to expire before a VM can be handed to you!

Some backstory on this: we've been working on a storage primitive to power incremental docker builds for Blacksmith's GitHub Actions runners. We realized early on that this was going to be a useful capability on its own since we notice a ton of our users with massive cache restore/save times in their workflows.

Fun fact, we decided to call it "sticky disks" as a result of this old HN comment thread: https://news.ycombinator.com/item?id=39957753

> I'm unwilling to use graphite if I can't convince the rest of my team

OOC, what are the main objections against?

That issue is from 2020 and has already been addressed. The fact that io_uring support is not "GA" is mostly a semantic aspect that doesn't affect most use-cases.

> Firecracker didn't have a very compelling I/O story

Can you provide any sources for this claim? We're running Firecracker in production over at blacksmith dot sh and haven't been able to reproduce any perf regressions in Firecracker over CH in our internal benchmarking.

We're noticing our development instances (which run on Fly.io) have been down for over an hour.

I'm obviously biased here but this is what we do at blacksmith dot sh. We run you GitHub Actions on consumer grade desktop CPUs with high single core performance, all inside ephemeral Firecracker VMs. Give us a shot!

Well, the point was that if 4 concurrent `git push`es saturates up to 1500 vCPUs then you'd need 16 of those hetzner dedicated servers (which you have to manage the uptime for) that you're paying for for the entire month. ~4 pushes is a very small amount and an org with a few dozen or so engineers will regularly see peaks higher than this.

Additionally, you'd have to ensure some isolation across your test runs (either by running the test suites in ephemeral containers, or VMs) which is additional engineering work for something that isn't business critical.

> we buy one or two very expensive machines that run CI and builds

This unfortunately does not work for orgs that have, say, more than 20 engineers. The core issue is that once you have a test suite large enough to have ~30 shards, you only need one engineer `git push`ing once to saturate those 1-2 expensive machines you've got sitting in the office.

The CI workload is quite amenable to "serverless" when you get to a large enough org size, where most of the time you actually want to pay nothing (i.e. outside your business hours) but when your engineers are pushing code, you want 1500 vCPUs on-demand to run 4 or 5 test suites concurrently.

A previous org I worked at had a python script (written by a PM) to download our github actions job history and shove that into an excel sheet. How this is still not offered by GitHub actions is beyond me.

Obligatory comment about India taxing air conditioners as luxury goods while people are literally dying of heat strokes every summer

The other side of this spectrum is that the founders spend months of their time fundraising rather than actually building the company.

This is simply false. For starters, GitHub actions by default run on Intel Haswell chips from 2014 (in some cases). Secondly, hardware being faster doesn't obviate the need for caching, especially for docker builds where your layer pulls are purely network bound.

A subtle challenge with "sticky disks" is that it requires your workflow steps to be idempotent beyond the point of "resumption", which can be tricky in a lot of cases.

GitHub has perverse incentives to not fix this problem because they charge customers based on usage (by the minute), so they make more money by providing slower builds to end-users.

Did you consider using a local (in the same VPC) docker registry mirror perhaps? https://docs.docker.com/docker-hub/mirror/

how do you ensure isolation between runs on a self hosted runner that way?

(blacksmith co-founder here)

it's unfortunate the amount of expertise / tinkering required to get "incrementalism" in docker builds in github actions. we're hoping to solve this with some of the stuff we have in the pipeline in the near future.