A lot of people have said that our brains are wired differently. I highly recommend Delusions of Gender by Cordelia Fine, it's looking into exactly those claims:

http://www.cordeliafine.com/delusions_of_gender.html