Building complete, optimised binaries can be much more complicated than just linking in a SO/A file. Things like cross-language LTO and PGO can be massive for performance and require integration throughout the build system.