You can. But this line of argumentation is usually used as an excuse to justify a total lack of diversity. If we lived in a world where people didn't routinely abuse these kinds of arguments that way, then it'd be a fair argument to make, but as it stands odds are firmly in favor of assuming that people using this argument probably has an agenda that has nothing to do with wanting diversity of thought.