The definition of stuttering involves having problems with physical speech. The definition of woman doesn't involve having problems with physical speech.
Stutterers are underrepresented in most conferences; even if you could get them up on stage, it would be a difficult presentation to listen give eloquently (and probably hard to listen to). Women are underrepresented at conferences, but it's not because they would have a problem presenting. Some people might have a problem listening to it, but that'd be largely because of their own issues.
Comparing stutterering - a physical (or perhaps mental) impediment - to females is insulting.
I've seen few female presenters bomb, and the ones that did - there were much easier explanations - no or little experience public speaker, didn't understand the topic, etc. The ones I've seen who know the topic and the audience and have spoken before in public are fine - no better than men. I do have occasion to listen to male speakers who stutter, and while I know they're smart people - they know their subject well - they can not present.
Being on stage in front of an audience who are tweeting about your physical attributes, not what you are saying; who are snarkily commenting about you being the token blonde; who are assuming you got there because of your birth, not your brains -- that's not a hindrance to speaking?
Even if none of those things are true, they can be running through the speaker's head, just as a stutterer is incredibly self-aware of their impediment. We create our own barriers.