I think there are different concepts of "cost" and "expensive" at play in this discussion which can lead to this kind of misunderstanding.

Having the population located in dense urban areas is more efficient and therefore cheaper in terms of the real resources that are required: infrastructure is cheaper to build, distances are smaller so that less time is wasted commuting and transport is cheaper (not to mention that highly efficient public transport becomes viable).

So in real resource terms, urban sprawl is more expensive.

But since land is still a limitation and zero-sum, competition can drive up housing costs in attractive dense urban areas to the point where it more than cancels out the efficiency gains of the urban density, and the cost of living in the urban area to the individual becomes higher than the sprawl.

This is why some sort of collective planning is required to make sure that as much as possible of the efficiency gains of urbanization are actually realized. This means having proper infrastructure mostly in terms of public transport.

(Of course, the other part of it is that you need to be careful not to look at just housing costs in the first place. For example, I don't even have to pay for a car, and I benefit from city infrastructure in many other ways.)

The land that can be used as "dense urban area" is strictly upper-bounded by zoning, precisely because it is so economically efficient: a few suburban homeowners cannot hope to compete financially with the ~400 potential high-rise condo owners vying to live on the same block, so instead they fight (and generally win) in the public-policy arena.

For the efficiency factor to overcome the desirability factor, we would have to actually issue the building permits to allow thriving urban areas to overtake some of their surrounding lower-density neighborhoods.

We've decided as a society that this is unacceptable, so urban areas will probably maintain their premium indefinitely. In a sense you're right. But only for self-inflicted public policy reasons.

> Until the claimed efficiency improvement can be translated to an actual day-to-day cost improvement

I don’t own a car, offload a bunch of concerns into City infrastructure, walk to a corner grocery store, and go from idea to meetings inside a single day, thereby letting me dance circles around competitors who have to schedule a call or flight out.

Cities don’t make sense for everyone. But they do for a lot of people. They make so much sense for enough people, that those people will pay a premium to have everything else nearby.