You wouldn't make this point about most other unchanging factors (i.e. genetics) that affect performance either. "Fairness" in high end sports is extremely subjective, and hard work is only a small part of how successful you can be, with time (which usually equals to wealth) and genetics being the most important.
"It's already unfair so there's no reason to care about it being unfair" is not a good argument.
the relative increase in performance would not displace anyone with a shot at winning.
Without data we have no way of knowing that. If that's true, we can prove it and put to rest the controversy.
I refuse to acknowledge this as controvercy that requires this kind of addressing.
That's certainly a large percentage of the motivation, granted. I do think a salient point is that there's nobody lobbying to make it unfair in some other way. If there were groups successfully lobbying to, say, abolish weight classes or allow the use of some expensive performance enhancement like a heads-up display, people would also oppose it.
If we can look at the data and say that transitioning and playing against women doesn't result in a relative performance increase vs the average athlete in the field, that's a strong argument that it's not unfair to let trans athletes compete against women. If we look at the data and see that "men who perform at the 60th percentile level among men perform after their transition at the 80th percentile among women" that's an indicator there remains an unfair advantage and our medical technology for transition needs to improve before trans athletes can fairly compete against women.