If you're walking by my house, you're in public, and you have no expectation of privacy. (Actually, that's not true -- my house isn't visible from the street, so if you're on my cameras, you're definitely on my property.)

If you're visiting me, you're on my property, ditto.

Any attempts to "fix" this will only make the world worse in the end. The people you empower to enforce your opinions about privacy in (semi-)public places will begin by exempting themselves from whatever well-intentioned rules you have in mind.