If my chip vendor decides to replace the ARM core on a SoC with a RISC-V core but leave all peripherals and memory mappings the same, I want to simply change the target and rebuild.
I mean, I don't know what distro you're working on (and whether they package these toolchains conveniently), but on Arch you just change your configured target in your conf/build system, and make sure you have the other one installed. If you're using autotools or cmake, you may not even need to change the configure scripts.
If you're doing embedded with memory mapped peripherals, surely you're building all your deps statically anyway.
If you don't have hardcoded addresses and are fully abstracted by a modern HAL and VM like on Linux then it's mostly a problem of the Linux distro or the OS vendor.
Clang isn't broken like that, and for clang you are right that the issue is normally dependencies. Again GNU's libc is kind of broken.
Cross compiling with Clang and Musl is not too bad.