This assumes that people reach "elite" status due to merit rather than to to complicated in-group favoring dynamics that serve to entrench certain class/ethnic/political interests. The more gatekeeping you allow the elites to do, the less meritocratic the elites are.

So the idea that some people are better at some things than other people is indeed true, but the idea that we can reliably measure that without the metric being gamed and corrupted is false.