Genetics is not that simplistic. A male born with androgen insensitivity syndrome have every genetic marker for being male, with male chromosomes, but get female genital and female secondary sexual characteristics. AIS has a range from mild, partial and full, and is only one of many areas where you get gender variation.

There were a study that looked if biological sex difference in the brain matched the chromosome or the gender that a person say that they are, and it found that biological sex difference did not match the chromosome for transsexuals. It matched the gender that the person said they were. The study also concluded that hormone replacement was not a factor as they only tested those who had not gone through that.

This mean that genital and secondary sexual characteristics are not adequate to correctly identify the genetic sex, nor is biological sex difference such as brain composition, bones or other traits which normally differ between genders. DNA is the closest thing to a clean binary state, but then we have strange cases where some individuals have multiple DNA in them, and thus have both XX and XY. As such the best scientific conclusions is that both gender and sex is a probabilistic result of the combined traits that we associate with male, female, man and woman.