this isn't an argument for accepting any old talk so long as it fills some demographic quota. It would be a strange conference that replaced quality of ideas with demographic diversity in their talk selection criteria. Rather, it just means conferences have more than one criterion to optimise for. Multi-objective optimisation is complicated and involves tradeoffs, but "all or nothing" is unlikely to be optimal.
Hopefully we can agree that a conference which had interesting technical content and was welcoming to a broad range of people would be better than a conference that had only one or the other.
The jist of my opinion is this: The subject technology should always be the highest priority at a technology conference.
If you consider the feeling of being, say, the only person of color at a conference to be a "superficial hangup", then perhaps your beliefs about what makes people comfortable have not been closely examined. It sounds like a statement about what makes you comfortable.
Perhaps there's a desirable balance to be struck here.