In a free market, no actor has the ability to 'let housing be crazy expensive'. If you don't build, someone else will and they capture whatever profit is there. This is the core of how the free market works.

People can only 'let housing be crazy expensive' when they can prevent others from supplying the market, which is only possible by wielding power, usually through the government. Which is what we have now.