But when someone asks me for money on the street, I almost always respond to them by saying "Sorry" or similar. I've always felt it rude to just walk by ignoring their presence. And probably over 95% of the time the person answers back something like "ok, have a great weekend" or "Thanks anyway".
That said, I think many people that walk past without making eye contact do so not because they think the person begging is inferior, but likely because they're hiding in their own shame.
I've seen similar behavior on crowded subways, where able-bodied people avert eye contact with the pregnant lady or senior citizen who is standing, wanting both to keep their seat but hoping someone else gives up their own seat.
At least for bigger outfits, it's easy to find out how your money is spent.
Short anecdote - A friend met a homeless man on Haight St who in exchange for a burrito showed him some of his wild world. The man ended up taking him to an abandoned warehouse covered with mattresses and needles. The man opened up a bit and told my friend that when he asks for money, he doesn't care so much if people don't give it to him. But what gets to him after a while is how most people won't even acknowledge his existence when he asks. He said he understood if people said no, but he appreciated at least being told no or even simply looked at.