If you are allowed to say, "our budget isn't adding up today so some people will have to stay on the wait list a bit longer," you aren't acting like it's a human right. Human rights are maximum priorities, not just high priorities.
An example of a reasonable human right would be that your own government can't kill you without a fair public trial. If a country ended up in a situation where they had to assassinate a businessman to keep their budget balanced, it would be reasonable to say "forget about the budget, you can't just take one of your own citizens out." However, if you had to loosen up the housing program because your tax revenue came in low that year, doing so would be potentially a reasonable option. "Potentially a reasonable option" is not even remotely on the table for human rights violations, which is why housing can't be a human right.
Housing could be a right in that sense: the government shall make no law preventing the construction of housing. Of course, like all other rights, it would be balanced against reasonable government interests like ensuring that the housing is safe.
The most pragmatic objection to this would be that everything you do prevents the construction of housing - if you build a city hall, you are preventing an apartment block from being built on the same land. Taxes prevent the construction of housing because the reduce the funds available to home builders and buyers. Property rights prevent the construction of housing because they would prevent me from setting up a tent in your front lawn. Making housing a right is a wellspring of endless absurdities, unless you are willing to treat it as a "maybe" goal. Maybe-goals and human rights are disjoint sets.
>Of course, like all other rights, it would be balanced against reasonable government interests...
So, let's look at a real human rights example, being assassinated by your government because you are protesting something. The dictator might claim that they were balancing your life against reasonable government interests - the reasonable response would be, "no interest is so reasonable that what you did would be justified." That's the thesis of human rights, if anything is justifiable then you don't believe in human rights. It's not that complicated.
So, if you think that housing as a basic human right can be suspended for reasonable reasons, you either don't really think it's a human right, don't believe in human rights at all, or are proposing a distorted crazyland.
Life: people die every day in the US. In some cases, the government even kills them!
Liberty: We put people in jail
Speech: Incitement to riot; slander
Press: Libel; national security
Religion: If your religion includes e.g. human sacrifice you certainly won't be free to practice it.
And so forth.
The people building houses are working because they are paid and they prefer that to the existing social safety net.