You might want to try using Docker's cache mounts? https://docs.docker.com/engine/reference/builder/#run---moun...

I've been using them locally of late, and they're excellent for storing state that's not obviously identical across invocations of the same layer.