Gender is a social construct. But they can't tell that for physiological sex either. Humans have a whole host of secondary sexual characteristics, most of which are more important than genitals in determining how other people categorize them. But even ignoring all of that, chromosomes don't strictly determine what sex characteristics someone ends up with. AIUI a lot of this has to do with the presence or absence of a testosterone wash in-utero (which itself is supposed to be determined by chromosomes, but this doesn't always work).

There have been documented cases of women with XY chromosomes who not only exhibit all the usual female sexual characteristics, but even have working ovaries and have carried babies to term. In fact, while I don't have it handy, there was a story not that long ago about a woman with XY chromosomes successfully having a baby, and then they discovered that her mother also had XY chromosomes.