I tend rock a similar look to Stallman and the sad reality is that people don't look at you favourably either way. I frequently get odd looks by people when out for a walk and I get stopped quite a lot by the police (I am White and live in the UK).

Simply putting on a clean shirt, some proper shoes and having a shave does wonders in this regard.

That is the reality of it unfortunately.

EDITED for clarity.