Think about the case where you have 32 CPUs, and so make starts 32 of the first tasks in the dependency graph. 31 of those tasks are routine compilations, but 1 task computes some data that a ton of other compilations need. What make and the OS don't know is that task would be 32x faster if was using all 32 CPUs. So by starting 31 random compilations, it's actually increasing the end-to-end latency dramatically, because so many things depend on that 1 job running as quickly as possible.

make and the OS basically only solve one tiny part of the problem of fast builds; maximize cpu utilization right now. They don't have enough data to make the latency of the build faster. It's very much a "good enough" approach; lots of people have faster builds because of "make -j32", but it could hardly be considered general enough to guarantee the fastest possible end-to-end build time. (And even things like Bazel don't quite have enough information to do this. Imagine you have some test that does "sleep 60" in the middle; you really want parallelism to go up to 33 when the test enters that state, and you want to ensure that you start it 60 seconds before the rest of the tests will be finished. There isn't enough data in the BUILD file to do that, though, so you're just stuck.)

Anyway, one thing you can do is get more memory and set -j higher than the number of cores/vCPUs. I think the kernel does a good job ensuring that each job makes progress, so if you have a scenario where something like a test benefits from starting as early as possible in the build and then basically goes to sleep, you'll still use 100% of your CPU, but make progress on that wait. The downside is that these things use a lot of RAM. I remember trying to build Envoy on a 64 vCPU machine once; each C++ process used over a gig of RAM, and I simply didn't have enough. I got more, and found that running 64 parallel compiles with SMT was about 10% faster than 32 parallel compiles. But, requires double the RAM (think it was about 80GB).

TL;DR lots of constraints to balance and optimize. Make is probably going to be the worst performing build system that you can use, but the field is hardly a "solved problem", and further research could probably yield general improvements. Also, buy yourself a lot of RAM when it's cheap. Having some code paged in and ready to go when the CPU is free is good for throughput.