The fallacy behind arguments like this is that when someone says that we're "proposing to push people to speak at conferences who aren't necessarily the most knowledgeable speakers" it is implicitly assumed that it is known how to rank the speakers, and that doing so is easy.

In practice, however, while one can often tell a really bad speaker and a really good speaker apart there are plenty of bordeline cases and the variance is big, so using minority-biased criteria to "break ties" will not necessarily lead to worse speakers on average, as long as these criteria are uncorrelated with actual skills.

(and I'm not sure they are, as the mean quality of a woman-presented talk in conferences I've attended has been consistently higher than the mean man-presented talk)