I think most people who have a problem with the homeless, myself included, are speaking primarily about people living on the street and very obviously either A) addicted to drugs, or B) mentally unwell. Whereas it seems that people who advocate for the homeless are advocating mostly for people who have fallen on hard times and are struggling to get by.
FWIW, I don't think people who are anti-homeless really have much of a problem with somebody who is sleeping in their car because they're struggling financially. But we do have a problem with people who live out of their car because drugs and/or mental illness render them unable to participate productively in society.
It only takes one person to file a complaint to the police about someone living in a vehicle on their street. It is, de-facto, a crime in most cities.
Probably because when we see a visibly homeless person, they are also very obviously mentally ill. There are lots of "invisible homeless" - in the sense that they don't look homeless, because they're not pushing a shopping cart around, aren't nursing oozing sores, and don't smell like they haven't bathed in two weeks.
Last fall, I was walking in my neighborhood and saw a woman asleep in her car. She didn't look like she pulled over to take a nap because she was too tired to drive safely; she looked like she was living in her car. But she looked just like any other person you'd pass in the street.
So, you're right, but it doesn't mean that homelessness-due-to-poverty doesn't exist.