Undoubtedly it's all connected in a feedback loop. I guess what I'm simply saying is that the problem of unwanted touching "can be solved" (or at least, ostracized into obscurity), and yet the underlying sexism can persist -- and, in one sense, be affirmed ("Hey I'm can't be a sexist, I never would just touch a random woman) -- and cause the same negative impact: the lack of women participation in conferences and in the tech field (or rather, society in general)
There are many reasons you don't see women speaking at tech conferences. Some is because they're passed over as non-members of the boys club. Some is because they don't want to attend and be part of the environment. I've had far worse things happen to be at tech shows - there's one I avoid altogether after being sexually assaulted in a hotel room - but like you said, there's a reason women don't feel comfortable talking about. i don't want to be the "token" harassment case. I don't want to become known for that above anything else I may do. So do I speak up or do I shut up.
That's just as large of an issue as how do I get that creep from putting his hand on knee. Arguably, it's bigger.