There's been a number of great bits of research over the past few years arguing that our "strong" housing market is a primary cause of economic stagnation.

Let me count the ways:

(1) reduced labor force agility as you say

(2) high rent and mortgage cost soaks up money that could otherwise fuel consumer spending, creating a demand constrained economy

(3) high housing costs in major cities also contribute to poverty elsewhere by preventing people from circulating between high priced urban and lower priced rural areas

(4) high housing costs risk actually destroying centers of excellence like the SF Bay Area and New York by making it impractical for new talent to move there