FreeBSD builds using makefiles, and the BSD make is not the same as GNU make, especially for the sometimes-complicated scripting they have to do inside of them.
Adding cross-compilation to a complicated set of makefiles isn't impossible but can be some work.
Beyond that there are other issues... eg. The ELF binaries generated need to declare themselves as FreeBSD binaries. A C compiler set up to produce Linux binaries won't do that. Maybe also they depend on a specific compiler (FreeBSD is using clang lately). When you build FreeBSD you also build a compiler so maybe this part isn't such a big deal.
NetBSD has had this[0] for ~20 years (requiring only POSIX conformance from the build host). Think of it as a sort of first principle[1] in action. I wouldn’t be surprised if FreeBSD could achieve some success leaning on this work.
Another that the compiler may use platform tools (assembler, linker) to build binaries, and those tools they may need a library (say one for writing ELF binaries) that isn’t available. All of them can be ported, but that can be a lot of work for a limited number of users.
Moreover, those users typically have (access to) the target hardware, as they have to be able to test the end result. So, a ‘normal’ compiler may not be optimal, but still be good enough for them.
Because of that, you tend to see more cross-compilers the slower the target hardware.