What if conferences are accidentally excluding women speakers? Is that still a bad thing?

Yes, we should chose speakers based entirely on their qualifications. But we don't. The problem is that people somehow assume that the current status quo of speakers almost always being men is what you get when you chose based entirely on qualifications, and any change from that must make things worse.

This is not about adding women by introducing gender bias. This is about adding women by reducing gender bias.