This is true but if we work with this line of logic, we should just remove the gender split in sports and make it purely a game for the best of the best.

What this means is that most females will be removed from sport since they can't compete with the average male athlete. Society has spoken already and decided they want to see female athletes which is why many male only sports have started their own female only sides.

This is a truly unfortunate edge case where trans people have an unfair advantage on one gender side and an unfair disadvantage on the other side. Perhaps the only fair way to do things would be to have trans only teams but the numbers probably don't work out. Or to have trans people pick sports where physical strength is not at play like motor racing, shooting, chess, etc.