If you define sexual assault/harassment including term "undesirable/unwanted sexual advances" and accept attractiveness to be one of factors influencing evaluation of another person as potential mate, it becomes not that hard to see lower limits for actions to be labelled "unwanted" as varying based on attractiveness of advancing person regardless of gender or social setting.

This would directly translate to welcome a sexual advance from one co-worker but not an identical one from another based simply on attractiveness. Social setting (e.g. workplace) may drive the lower limit below skirting board for some individuals, but that does not make certain social settings universally irrationality free grounds where evaluation as potential mate has no effect on social (and even professional) interactions.

While both men and women are capable of rational thought, likewise both are not individuality, emotion and desires free.