It's not (directly) about testosterone in this case, women are more guarded/suspicious because they're being hit on all the time. For men it would be an opportunity, for them it's a nuisance/threat.