I don't see anything offensive in calling someone "a man" or "a woman". If I do that, and you respond by labeling me as a "transphobe", fine, but as far as I'm concerned you are responding with a pejorative (unflattering) term to a neutral one.