Indeed! I got dragged to the police station once by a guy cause I took a picture of his front door. He even tried to force me in to his car. Instead I walked to the cops with the guy driving next to me. He explained the problem and the cops told him to leave me alone.

The problem is that at least in the US there is a prevalence of "public access" places that are technically "private property" and enforce bans on photography as such, including public transport stations in some cities.

And malls.