Look at the 100m sprint. Of the top 25 (regardless of gender), as far as I can tell, none of them have two X chromosomes.
But none of them are Caucasian, either.
Why isn't there a large movement arguing that sprinters should only be allowed to compete against those of the same race? If I (as a white man) took up sprinting (ha!), it would kind of suck knowing that I don't seem to stand a chance of setting the world record just because of my genes. But it seems like white sprinters manage to deal with it.
And, well, if there were separate races for people of different races... that wouldn't really change anything, would it? The world's fastest sprinter would still probably be non-white, the only real difference would be that the world's fastest white sprinter would get to say "I'm the world's fastest white sprinter (but still not the world's fastest sprinter)". You're just giving out an extra trophy- everybody knows who's actually the fastest.
Why isn't this argument valid for sex separation in sports, too?
You'll find that most athletes at the top are genetic outliers. There is already some unfairness towards those with low testosterone. However if you allow people that have lived long periods of time with high testosterone to complete against women who have not I suspect you'll find very few of those women ever make it to the top.