Housing is not rising in price because it is an inelastic good. Housing is rising in price because the growth in the supply of housing is less than the growth in the demand for housing.
Indeed, the very fact that housing has inelastic demand means that it is particularly susceptible to price reductions when supply is higher than demand.
Lack of supply would cause prices to rise - but so would cartel behavior on the part of those who control that supply. Housing of the type necessary for relatively low-income individuals, where the location must be close to jobs (see above), is typically controlled by a set of entities who exhibit cartel behavior in how they price the rent.
Anyway, supply and demand are murky concepts that don't map well to reality when trying to take them out of the supply/demand chart. You can't actually quantify potential "demand" because "someone wants a good or service" is not a data point for demand, only actual trades that happened do.
I think you have to be careful with the definition of "unaffordable".
Housing, at this moment, is unaffordable in the sense that the cost of housing squeezes many people's discretionary budget, savings, and even sometimes the budget for necessities. It is not unaffordable in the sense that (most) people do have enough money that they can pay for it, even though paying for it might cause them hardship elsewhere.
Housing is an inelastic good - particularly for demographics who have limited access to transportation and therefore need to live very close to where the jobs are. The price increases until it consumes all the money available to pay for it.