If you are violating their boundaries, then yes. I don't know if you are aware of this, but people who have been assaulted may have a problem with people violating their physical space and you doing something that you consider harmless may actually be a big deal. The point is get consent and default to doing no harm, which is basic human decency.