Is that much of a problem though? For SBCs like the Raspberry Pi you can do builds on the SBC itself. While this is neat, it is far from optimal because builds are very slow. It's much more typical for embedded systems to do you build on a host PC with a cross-compiler.
Sometimes getting that cross compiler environment set up properly, especially with all of the dependencies and build tools, is more hassle than just letting the build run overnight on the SBC itself. Especially when it has a configure script that wants to check on various properties of the target architecture as part of the build.
It's already impossible to compile Firefox on a machine with 1GB of RAM because some of the files are just too big. Even with swapping enabled the build process crapped out for me last time I tried it. While 1GB machines are pretty rare these days outside of SBCs, you could easily blow out a 4GB machine if you tried to compile the whole damn thing at once, and 4GB machines are still commonly found.