How can that be worse than "there are millions of unoccupied places, most housing belongs to the same people who don't need it, and yet there are millions living in the street"

It's not just about you

the claim is who is doing "nothing".

Working, saving, and buying a house is doing a lot more than doing nothing and expecting something

Most houses are owned by people who do not work for it. It's the rent from their other houses that give them even more money, effectively doing nothing. Let's not think those people deserve our praise for their parasitism.