Someone who is panhandling wants money and they get very angry when you interrupt that process. At least in the US, the panhandlers tend not to be the genuinely homeless--they mostly tend to be scummy people. It's a semi-regular news occurrence for someone to follow a panhandler with a camera to find out they really aren't homeless.

It really makes the political discussion about how to help the "homeless" a lot harder because now you have to deal with "How do we make sure that we help the "homeless" instead of these scummy jerks?" And it's impossible to make the separation--the scummy jerks will always work your system better than the genuinely homeless. It's one of the reasons why I'm generally for the basic income kind of approaches--the scummy jerks can't work your system to the detriment of those who genuinely need it.

In the US at least, the "genuinely homeless" that you could actually help tend to be nearly invisible because they don't want to be seen--getting seen gets you noticed and harrassed.