That's entirely true if all houses are the proverbial spherical identical house, but people chasing to remain "where they are" also slows down upgrades and downgrades.

If a family that would normally have sold their older two bedroom and moved to a four bedroom instead chooses to remain in the two bedroom longer, that two bedroom doesn't appear on the market, a four bedroom languishes.

And if nobody is building two bedroom houses, that can have ripple effects.