Is basketball a conspiracy against short people? Weightlifting for tall people? Are combat sports full of weight-ist people who won't allow a heavyweight pound a welterweight? Is football the least ideological sport because it allows a greater variety for BMI and height?
Let us not bury the glaringly obvious fact that sex is a category with one of the strongest, if not the strongest, explanatory power for performance across almost the entirety of athletic endeavors in existence.
Height is a huge advantage in basketball. The average height of an NBA player is over 6’5”. Height is mostly genetic except for malnutrition. Different populations have different average heights.
The optimal build for a weightlifter is to have short arms and legs and to have the two parts be of equal length. That is mostly genetic. Of course different population groups have differences here as well.
Weight is an advantage in combat sports both because you can dish out more and because you can absorb more punishment. I suspect that high bone density is advantageous and is mostly genetic. Different populations have different bone densities.
No one is denying that sex has explanatory power. The point is that many other things have explanatory power as well and that choosing which categories to consider is ideological.
Why have separate sports for sex? Why have separate weight classes? Why have a minimum age requirement for gymnasts? Why not allow professionals until 1992?
Why not have separate height classes for basketball? Why not have separate weight classes for football?
No need to complicate; if we didn't have sex segregated sports, for almost all sports (save for maybe open ocean swimming), we couldn't have any women competing at all[1]. In fact there are "battle of sexes" type exhibition matches that demonstrate the rough placement eg Braasch vs. the Williams sisters.
[1] To expand this statement because I know some people will get offended; this doesn't mean men are better at sports than women. It says, at the tail end of distributions, where most of the professional sports are played, tail end of men's distribution will be better than tail end of women's, leaving no opportunity for women to compete if distributions are intermixed.
So unless you have hard statistics to back up these claims or are an expert in one of these fields, I would go with the experts' decision to sex segregate their leagues because they might know a few things or two we won't necessarily consider.
Just from my amateur interest; for outdoors archery, the draw strength matters a lot for accuracy, in curling likewise drive and sweeping strength make a difference.
To be completely clear, in some sports there appears to be a moderate risk that elite cis-women will be completely unable to compete with elite trans-women. This is going to be visible and apparent. I don't think this will be acceptable to the most people, and they will likely look to change the system.
Nit-picking your other points: there are not really any popular rich-people-only sports any more, except perhaps auto racing. I don't think broader society is all that interested in the egalitarian properties of yacht sailing. R.e. the clothing technology: yes, and those things were banned because of the advantage they conferred. That's exactly what I'm talking about here.
I can assure you, from personal experience, that rowing, cycling, tennis, and lacrosse are overwhelmingly wealthy sports. You don’t have to pass a credit check to play them, but they are absolutely a class apart (both economically and racially) versus sports like baseball and basketball.
Skiing is expensive for most people, but having grown up in a mountainous area, most kids could and did go skiing for very cheap.
Genes for height, genes for fast twitch muscles, testosterone, etc… aren’t the same for different groups. Some of these disadvantages are just as insurmountable.
Many sports are closed to poor people. Equestrian, sailing, etc… aren’t sports for the poor.
Richer countries often have better technology (see those Nike shoes and the swimsuits that were banned).
We are just in the land of competing interests.