The problem is excessive regulation, and excessive land prices, due to zoning restrictions.
- the wrong level of government has the power and the money. Housing is generally a local problem, but in many countries the cities are chronically underfunded -- higher levels of government have significantly more taxing power.
- misalignment between levels of government. Higher levels of government may be incentivized to increase housing stock. City governments are captured by NIMBY's.
- trying to do "easy" fixes. This is a hard problem; proper fixes are difficult and expensive.
- trying to preserve property values. People get really angry when their property values go down so governments try to have their cake and eat it to. They design programs that make it easier for people to buy houses. If those programs are successful, they increase house prices, making the problem worse.
Canada tried the "have cake eat it too" approach for 20 years and ended up with insane housing prices. The current government has finally adopted effective programs, addressing all 3 of your points. House prices are down this year in Canada; 1.9% overall, down 3.4% in the two most overheated markets.
What Canada has done:
- tying money for cities to zoning reforms
- vacant house taxes
- cut back on immigration
- directly building affordable housing
Notice that the above includes both the "right" and "left" proscriptions for reform. Both sides will claim that it's their proscriptions that's the effective one.
- Not building enough (= insufficient supply). The reasons vary per country.
- Too many people, mostly due to immigration (= excessive demand, with supply failing to keep up... in some cases, it's virtually impossible to meet demand).
- Competition from outsiders e.g. wealthier people from richer countries and/or large investment funds.
Some countries have all three problems, while others may have only one or two. However, it seems that many governments continue to misdiagnose the problem, placing too much emphasis on factors that in reality contribute very marginally to the problem. This may be due to incompetence or simply whatever agenda they have at the time.
If I had to make a wild prediction, I'd say the problem will continue to get worse until the whole system (not just housing but many critical parts of the country) eventually collapses in many places.