The problem is, women in tech isn't a topic I speak on, so by your logic, they should never have reached out to me in the first place.

No, it's not wrong in general. Don't try to make the argument about some abstract principle reduced to absurd generality. It was "wrong" in this specific instance to limit the topic to one narrow topic which is related to speaker's gender without letting the speaker to speak on more broad issues in which she's actually more knowledgeable and interested. "Wrong" meaning that if I visited a conference I'd wish the organizers didn't do it. Of course, it's their conference and they can make all the speeches be on any narrow topic they want, regardless of how displeasing it to me - I just say it would be better if they did not limit their horizons this way.