> If they are the subject matter experts, then yes, I would prefer they are all present, and none of them be excluded because we are searching for faux diversity.

My understanding is that the complaint is not "these men are not qualified", but "there are women who are qualified who are systematically being ignored".

There isn't an objective ranking of subject matter experts, from which conferences simply invite the top N people. Nor, as far as I know, is anyone in favor of inviting non-qualified minorities to speak. The question is "why are qualified [women, people of color, trans people, etc.] not being invited?".