I thought user-space applications should be fine? Bootloaders that kind of stuff has to deal with low-level interfaces and those do differ from SoCs to SoCs.

You can build multiple functions to handle different architectures by function multi-versioning[1], not sure if it works for embedded though.

[1] https://gcc.gnu.org/onlinedocs/gcc/Function-Multiversioning....