> especially since we're told that there's no actual difference between men and women?

Who is telling you this? This runs contrary to decades of research findings (however distasteful some might find that).