Of course that is true. That doesn't mean that the cost imposed on others is zero.
It's never zero. But you're making it out to be a significant cost relative to the effect on the person making the decision. Having a way to internalize the cost of others' displeasure at being aware of their possible misfortune wouldn't materially affect their decision one way or the other, because it's so much smaller a factor than the possible misfortune to begin with.
But you're making a fine argument for charity. If a thousand people don't want to see a person go homeless, it doesn't take much from each of them to make it so they don't have to.
And if 1000 is not a big enough number to make you think twice, then make it 10,000. Or 100,000. (The number of homeless in the U.S. is considerably larger than that BTW.)
You're comparing the entire US homeless population to one person. The actual ratio is less than one in five hundred.
The person taking the brunt of the hardship is clearly the person suffering it and not the person with the misfortune of having to observe them, and they also have the best incentive to take steps to prevent that from happening. Forcing choices on them doesn't fix it. Maybe it makes it so you don't have to look at them, but if the net result is sufficiently worse that they wouldn't have chosen it given the choice, you're not actually helping them, you're just making the suffering less visible.