Common grounds and uniformity - if every library were using it's own build system it becomes very hard to combine libraries. See c++ for example, when including a library you always have to either wrap the build, rewrite it to your build system or find a a way to prebuild. And every library is unique. Contrast this with other systems where you just add a line to your package.json file. Big red flag is that people come up with those header-only libraries or single .c-file only libraries, just to get around the problem of package management.

Language rules define dependency rules, that the build system is not aware of. For example transitive #include chains, requiring stuff like gcc -MMD to generate makefiles. This also improves IDE experience with suggestions on where to find imports instead of elementary error messages like "header file not found".