I think the counter argument is that housing prices will fall and be closer to free market. I can't really get much worse for the poor, the status quo clearly isn't helping them.

You do have a point, but there seems to be another side as well.