You have to consider the power imbalance due to gender norms. In our culture, men typically pursue women romantically. This is not always the case, but it is the norm. Hence, in a social setting where it is acceptable to pursue someone romantically, a woman complimenting a man's appearance has the implied meaning "I would be receptive to your romantic advances." [1]

However, in the same social setting, when a man compliments a woman's appearance, the implied meaning is "I am pursuing you romantically." And this is fine in social settings, otherwise our species would die off.

But it's not fine in the work place. You feel that a woman at work giving you such a compliment is "harmless fun" because if it was a normal social situation, you would have the power: it's up to you to pursue. But for many women it is not "harmless fun" because it's similar to the social situation, and they feel romantically pursued, which is not okay at work.

[1] Please note I do not consider this a promise, so let's not confuse this with "asking for it." Human communication is subtle, and such implications are how consensual "courtship" happen. Of course, once there is a rejection, whatever implications one thought were there don't matter.