This feels like the way, but comes with the huge caveat that it doesn't work with bazel remote execution (unless you can rig up something to sync your nix store to every remote builder).

I haven't used it personally, but the Nix manual mentions support for dealing with Nix stores in distributed builds https://nixos.org/manual/nix/unstable/advanced-topics/distri...