Having a right to housing is not the same as having the right to "living where you want".

If they actually had that right then a Spaniard would have the right to live in the king's and politicians' houses which is obviously not a right.