Materials aren't the largest cost around here (California) There has been a massive worker shortage since 2008, so even incompetent people can charge way more than material costs for most installation jobs. (And then you pay a second person even more.)

Permitting, engineering, etc are also expensive.

Finally, construction costs are difficult to estimate up front, so it is difficult to be a rational actor when deciding to build or not.

After all, if you hit a 33% cost overrun half way through building, your best option is to pay the 33%. At that point the house costs 83% of the expected price of starting over. If you stop now, then you are out 50% of the initial budget.

If you think facing such a tradeoff is rare, look around for lots that are for sale with pre-approved plans or partially constructed houses. They're usually pretty easy to find.