Two interesting aspects of your argument here. One, you've quietly dismissed any possible sexism. And two, after accurately pointing out a problem, your solution is basically, "well fuck them, then." Nice.

I've been involved in selecting conference speakers, and we went out of our way to find geeks who aren't used to speaking up. Quiet people often have the most interesting things to say.