It makes a huge difference.
For what it's worth, I'm working on a project with about 1000 dependencies and incremental debug build time is about 4 seconds on my laptop.
That's already pretty good.
Number of dependencies likely won't affect incremental build times except for linking and replacing it might offer some good gains for incremental builds.
Default linker is quite a bit slower.
And on my Macbook Air M2, where I actually develop, these things happen fast enough to call them instant. Perhaps I'm a bit spoiled there due to the excellent hardware. As a comparison, a Typescript project I'm working on using a more powerful Macbook always takes about 5-10 seconds to build.
I don't doubt that actually large Rust projects take a long time to build, though, but even these small and mid-sized were rather slow to build a few years ago.
You can follow my Dockerfile of my project as an example on how to do it
Also, these days it's possible to use cheat codes aka ChatGPT to flounder through almost all the difficult Rust problems that might have been show stoppers a few years ago. It's looking pretty great on that side of the fence.