I think you're right in the sense that people, when given some kind of muscular policy or attitude, don't really care about gender per se.

Only when nothing else is on the line will they say "I won't vote for a woman."