Certainly, it's possible one can be accused of anything, with or without any basis in reality.

I don't believe the risk of being accused of inappropriate behavior is anywhere near what you seem to, and accordingly have spent many, many hours and evenings socializing with colleagues, many of whom (men and women) remain friends long after the job is over.

Govern yourself according to what you deem appropriate, but suggesting that it is a fact of society that women are untrustworthy is neither true nor useful to society at large.