There's much more to this... A totally attractive guy (model-like) that emanates low self-confidence (shyness, bad posture, ...) could still have his comments perceived in a negative way (i.e. as being sexist, not compliments). So, I believe that we tend to judge this from the perspective of the perceived intent of the person making the comments. A self confident gut -> he's probably just being sincere -> cure. A not-self-confident guy -> he's trying to get something from her -> looser. (Probably applies with the sex roles reversed, but I have no actual experience to lean on).

At the workplace, the label looser is just switched for the label creepy/sexist.