Trans women have been allowed to compete in the Olympics since the 80s with some stipulations around testosterone levels being maintained as very low. It would already have been a problem, if your simplistic theory were true.