"If your only concern with a conference is about the panel diversity then perhaps the conference is not for you."

This sounds like a straw man argument. You've created the fictional idea of a person whose interest in a conference is solely the diversity of the panel, as if there are people who don't even care what the conference is about or have any idea of the subject matter; they just like seeing different kinds of people on panels.

well, this is exactly what I get from all those who are complaining. To me, they seem more interested in sex/gender/race of the speakers than WHAT is going to be presented.

Squeaky wheel gets the grease. These people are clearly willing to complain. No complaints about what is being presented implies only that they are satisfied with what is being presented; it does not imply that they do not care about it.

Except that the point of the conference was to present the audience with itself as a community. If the conference was only about the talks, then they might as well just make a YouTube channel and ask people to send them videos of talks.