You can cache the Rust target and make it use Rust's cache properly.

There are two common ways i'm familiar with. Cargo Chef and Manually. Manually can be a bit involved, but you basically just build with empty source and prune your fake-lib files from the cache dir. Cargo Chef is very easy.

My docker images build as fast as my local images. Though if you have many crates in a workspace it can start to become difficult.