You might want to try the actions cache "--cache-to=gha --cache-from=gha", but still it needs to pull that stuff down, just that locality is likely better here.

There's also an action out there "GitHub cache dance" that will stick your whole buildkit state dir into the gha cache.