So far cargo doesn't do binary dependencies, so when you start a project, or do CI/CD, it compiles the all world from scratch.

I'm not intimately familiar with the internals of rustc, but my impression was that everything gets compiled and dead code is later eliminated. In order to avoid compiling unnecessary stuff, you generally need to use feature flags (e.g. you can say "only compile this if the serialization feature is enabled").

Actix, Diesel, Serde and their dependencies are definitely most of build time.