Height is a huge advantage in basketball. The average height of an NBA player is over 6’5”. Height is mostly genetic except for malnutrition. Different populations have different average heights.
The optimal build for a weightlifter is to have short arms and legs and to have the two parts be of equal length. That is mostly genetic. Of course different population groups have differences here as well.
Weight is an advantage in combat sports both because you can dish out more and because you can absorb more punishment. I suspect that high bone density is advantageous and is mostly genetic. Different populations have different bone densities.
No one is denying that sex has explanatory power. The point is that many other things have explanatory power as well and that choosing which categories to consider is ideological.
Why have separate sports for sex? Why have separate weight classes? Why have a minimum age requirement for gymnasts? Why not allow professionals until 1992?
Why not have separate height classes for basketball? Why not have separate weight classes for football?
No need to complicate; if we didn't have sex segregated sports, for almost all sports (save for maybe open ocean swimming), we couldn't have any women competing at all[1]. In fact there are "battle of sexes" type exhibition matches that demonstrate the rough placement eg Braasch vs. the Williams sisters.
[1] To expand this statement because I know some people will get offended; this doesn't mean men are better at sports than women. It says, at the tail end of distributions, where most of the professional sports are played, tail end of men's distribution will be better than tail end of women's, leaving no opportunity for women to compete if distributions are intermixed.
So unless you have hard statistics to back up these claims or are an expert in one of these fields, I would go with the experts' decision to sex segregate their leagues because they might know a few things or two we won't necessarily consider.
Just from my amateur interest; for outdoors archery, the draw strength matters a lot for accuracy, in curling likewise drive and sweeping strength make a difference.
Is basketball a conspiracy against short people? Weightlifting for tall people? Are combat sports full of weight-ist people who won't allow a heavyweight pound a welterweight? Is football the least ideological sport because it allows a greater variety for BMI and height?
Let us not bury the glaringly obvious fact that sex is a category with one of the strongest, if not the strongest, explanatory power for performance across almost the entirety of athletic endeavors in existence.