Compiling them in release mode does have an advantage if the proc macro is used a lot in your dep tree, since the faster invocations compensate for the increased compile time. Another option is shipping pre-compiled macros like the serde maintainer tried to do at one point, but there was sufficient (justified) backlash to shipping blobs in that case that it will probably never take off.
Here's a comparison of using serde proc macros for (De)Serialize impls vs pre-generated impls: https://github.com/Arnavion/k8s-openapi/issues/4#issuecommen... In other words the amount of code that is compiled in the end is the same; the time difference is entirely because of proc macro invocation. 5m -> 3m for debug builds, 3.5m -> 3m for release builds. It's from 2018, but the situation is largely unchanged as of today.
And the majority of procedural macros don't produce that much code and like you said their execution isn't the biggest problem.
E.g. the recent article about a db system ending up with 30?min compiler times and then cutting them down to 4min was a case of auto generating a whole (very enormously huge) crate (no idea if proc-macros where involved, didn't really matter there anyway).
So yeah, kinda what you said, proc macros can and should be improved, but rarely are they the root cause.
There are other legitimate criticisms you can raise at the Rust borrow checker such as cognitive load and higher cost of refactoring, but the compilation speed argument is just baseless.