Is it somehow impolite, or demonstrably incorrect, to assert that there are people who are in fact stupid? Or people who are quite functional but not particularly bright? That there are people who, through no fault of their own, could never achieve certain intellectual goals no matter how much time or effort because they are simply not mentally capable?
Life experience tells me otherwise.
What I don't buy is all people being able to do the same degree of mental work in the same time, which is basically what the PhDs for all stance seems to require. If it would take someone 30 years instead of 5 to learn the stuff required to finish a PhD thesis, it's not practical for that person to pursue a PhD.
The developmentally disabled, stroke patients and others are unignorable cases of people who need much longer than others to learn things. It's a pretty big stretch to assume that the learning speed would be so close to same for all the rest of the people that there wouldn't be any significant fraction of people who learn so slowly that pursuing a PhD isn't a practical option for them.