I've seen (sometimes drastic) performance improvements when switching to LLVM's new linker lld[1]. It works quite well on both Linux and Windows in my experience.

[1]: https://lld.llvm.org/