There are ± tolerances at every interface, so if you're off a bit on the foundation, you can balance things out in the rough frame, so by the time you get to finish framing things are pretty square/plumb/level.
But if the pre-built stuff is ±0, then there's no wiggle room in that part of the build, so the rest of it has to be that much tighter as you've not nowhere to adjust things.
The problem with housing in America always comes down to the way it was financialized and securitized: too much relies on "line go up, forever". There's no room for new blood/capacity (read:competition), there's no room for "investments" to lose value.
Nobody is going to spend $x on a brand new house without having some say in it, and so those houses tend more and more toward "high end/luxury". After all, why go through the hassle of all the paperwork and building and NOT sell for the highest price you can get?
Same thing happens with cars; the market for car buyers is much larger than the market for new car buyers, but only new cars ever get made. Nobody is making used cars, or even the absolutely cheapest possible, which affects the whole supply.
https://www.noahpinion.blog/p/market-rate-housing-will-make-...
That's not to say housing construction regulation isn't a problem. It is, but two things can be problems at the same time.
Well, duh!
As mentioned in this comment[0] land and labor are still the dominant part of the costs. So if municipalities would allow and prefer denser housing cost would be lower.
And of course if we are already talking about quasi-standardized (cookie-cutter) units, then there's even more reason to scale up projects so prefab components could be shipped in. (Though of course we again run into the tragedy of small scale. Metro areas are made up of too small suburban cities, they don't want a big project, they don't have the infrastructure for it, they don't want the extra traffic, and so on.)
You can build a tract of homes in 90 days (or less) using stick frame. They won't be fancy, but they'll be up to code (and with a little extra cost (<10%) and effort, they can be made much more efficient than just code).
A crew of 4-5 guys (plus some subs) can build a custon home in 80 days:
* https://www.youtube.com/watch?v=LYd73YP57Ik&list=PL8XEQ1XKYN...
If you pipeline that and use a standardized, cookie-cutter plan an entire row can be build out in a similar time frame. Production builders do it all the time:
* https://www.newhomesource.com/learn/custom-or-production-bui...
There are of course mass-manufactured elements that can speed things up, like using trusses. There are also now services for pre-cut framing that saves on-site effor of measuring and such:
* https://www.youtube.com/watch?v=o2FdAdxjSpw&list=PLDYh81z-Rh...
The building of the structure is not the bottleneck, it is the approval process and NIMBY road blocks that can add 2(+) years to a project.