These tests are predictive, and as you point out, it's disingenuous to claim otherwise. They're measuring something valid. However, they're doing so relatively poorly, or rather, roughly. Even the quote you cite suggests that the total amount of variance accounted for was about 19%, which isn't too shabby by any means, but is also very crude when you think about implications for real-world consequences. Interpreting that 19% also becomes very murky, when you consider factors like teaching to tests, that tests like those are used for academic and career selection, leading to self-fulfilling prophecies. In rare circumstances where researchers have been able to remove some of these effects (for example, due to unusual legal decisions about acceptable use of test scores), the relationship between test scores and career performance has been shown to be very weak in the short-term, and almost non-existent in the long term.
Somewhere underlying the distribution of test scores is some distribution of ability that makes it easier for some people to thrive in certain settings. But ability is just one thing in a long list of things, along with practice, resources and other environmental factors that contribute to success, and sociological dynamics. Add to that test scores being very crude indices of ability to begin with, and you're in a very messy situation.
What's frustrating to me is that there's this sort of all-or-nothing attitude about testing, where tests are seen as useless, or as a perfect indicator of skill or ability. The truth is in-between.
This is 19% after range restriction. They're making the same point that the graph of quartiles does: even after you set an extremely high bar, differences in the test score are still predicting quite a bit of variance despite all the other possible diluting factors like geography/family/test-error/personality/interest/wealth/opportunity... If anything, it shows that institutions aren't being 'crude' enough - if they were using the test scores optimally and extracting all of the signal, the variance would be 0%.
Yes. This is a major point people need to focus on.
I think, to bastardize a famous quote, that testing is the worst simple measure of aptitude, except all the others we've tried. In a perfect world, any admissions/hiring decision would be made after multiple independent in-depth examinations of a candidate's past work, plus extensive interviews. Since that level of vetting is unfeasible for all but a tiny sliver of the most desirable spots, we need ballpark estimates like testing that will be more predictive than coin flips.
I also spent some in summer camps with other students in the study and have remained casual friends with some of them over the years. I don't see any evidence to suggest that that level of demonstrated ability is transient.
I have never understood the intention behind this type of statement (which I've seen many variants of commonly espoused). Let X be the event "got a high test score at a particular test at a certain point in time." Is Matthews saying X = X (an uninteresting tautology)? Or is he claiming that X is uncorrelated or not predictive of other variables of interest (career success, ability on other task) which is clearly empirically false.
This depends on just how specific a skill is, whether or not the particular subject is a polymath or specialist, and much more.
To take an example from athletics, Lance Armstrong was a world-leading cyclist (with, it turns out, the advantage of doping). When he entered into marathon events -- for which his light body frame is generally still an advantage -- he placed respectably, but not as one of the very top finishers. A skills or abilities test for Armstrong had to be specific to his ~60 km cycling distance specialisation.
There is some crossover of many skills, but the deep specialisation, ingrained structures, algorithmic knowledge, solution sapces, bags-of-tricks, etc., that are associated with being best in class (not merely "very good") can be quite complex.
The way I think about "intelligence" is not to mull over what it "really means" but to think about lots of common "intellectual" tasks (in which aptitude can be measured) that people are interested in: reading, writing, math, programming, solving puzzles, etc. Or ideally, theoretical "potential" in each of these tasks, since one person may put more or less effort into another field.
What's supported by research and anectodal evidence is that if you did principal components analysis on these measured "aptitudes", there would be a principal component explaining a lot of the variation. A good test can try to measure this principal component.
But when you're talking about the very highest levels of specific tasks like cycling or swimming, it would be reasonable to expect that amongst the population of already "elite athletes" there would be less of a dominant first principal component
There are different definitions of intelligence. One I like (though I don't quite find it comprehensive) is David Krakauer (of Santa Fe Institute), "Intelligence is Search". That is: it's a search across an n-dimensional space for a least-cost function.
There's the question of "function for what?", and I'm thinking here something which allows for useful prediction or outmaneuvering an adversary (as with chess).
This gives a set of related or subroutine functions which are required, including many on your list: information acquisitionprocessing, and output. An interesting concept I ran across in the past year or so is that of the medieval curriculum, comprised of the trivium and the quadrivium. The first is further broken down into grammar, logic, and rhetoric, or if you prefer, input (and framing), processing, and output. This tickles my brain.
Another element I'm familiar with is that there comes a point in any skill at which it's somewhat less the things you get right as what you avoid getting wrong that matters. A champion athlete has strength or stamina, but also optimised physiology (Michael Phelps's long arms and torso, short legs, and huge feet, for example, are swimming gold, QED), and not screwing anything up in the event. Nailing starts, turns, finishes, and technique, not getting in your own way.
Further pondering intelligence also brings into the fold what knowledge, wisdom, correctness, and quality are. Interesting questions.
1. any doctorate
2. STEM publications
3. STEM doctorates
4. Patents
5. Income in the 95th percentile
A person in Q4 was 60% more likely than a person in Q3 to earn income in the 95th percentile. I'd say that's pretty significant.
[1] https://www.scientificamerican.com/sciam/assets/Image/Child_...
(I don't know enough about #4 or #5 to make a comment.)
That is, intelligence is a necessary (but not sufficient) condition to get a graduate degree in STEM. The other necessary conditions you mention (training, interest, etc.) are also important, but still require a baseline ability.
Even looking at undergraduate degrees, IQ is a significant factor in what degree is obtained. Physics and astronomy majors have an average of IQ [1] of over 130 -- that's 2 standard deviations above the mean. If interest, training, and hard work alone were sufficient to get a physics degree, we'd see a much lower average IQ among physics students.
[1] http://www.randalolson.com/2014/06/25/average-iq-of-students...
Everything you wrote just now agrees with my statement.
Originally you wrote "All five rely heavily on intelligence". It is that emphasis on "heavily" that I am disputing. I argue that it's less important than training, though of course not un-important.
I disagree with your interpretation of [1]. You wrote "Physics and astronomy majors have an average of IQ [1] of over 130". However, your [1] shows not a measure of IQ but an approximate estimate of IQ based on the cumulative SAT.
After showing the plots, your [1] then points out the correlation is only with the quantitative/analytical section of the SAT, and not the verbal, commenting "we see why this IQ estimation is potentially misleading."
> This tells us that the original plot is actually showing preference for quantitative majors: The higher the estimated IQ, the more quantitative/analytical the major, and the fewer women enrolling in those majors.
> This brings up an interesting question of how valuable the SAT is as a standardized test across all majors, if a higher SAT score is really only indicating that the student is better at solving quantitative/analytical problems. Not all majors require a high analytical aptitude, after all.
Your source therefore does not seem to be making the statement that "Physics and astronomy majors have an average of IQ of over 130" but that physics and astronomy require a high analytical aptitude, which is correlated to how well a student does on a test for analytical aptitude, which is correlated to the cumulative SAT score, which is correlated to g factor. That's a weaker conclusion than what you described.
In any case, undergraduate study does not require the ability to identify what is scientifically interesting, which is a key point of my objection to your original statement.
EDIT: Also, your [1] points out "men and women have about the same IQ" then shows that physics and astronomy has a greatly disproportionate number of men than women. Thus, being male is a stronger correlation for those undergraduate degrees than intelligence. Thus suggests that some other factor than general intelligence is biasing the numbers.
That is:
P(high IQ | high test score) > P(low IQ | high test score)
What you're trying to say is that getting a doctorate a degree doesn't depend on intelligence, and is just an indicator of some "group bias". What I'm saying is that test scores are correlated with IQ. This implies that if you do not have high intelligence, it is going to be extremely difficult to get the test scores required to get into graduate school, and thus intelligence is required for admission.
In other words, intelligence is indeed a necessary but not sufficient condition to get a graduate degree in STEM.
The question is, how do you know that getting a PhD relies "heavily on intelligence"? How do you know that intelligence has a strong or weak correlation? Or that there isn't some sort of threshold, where an effective IQ of at least 90 is important, but 140 is no more important?
What other factors have you considered, and why have you concluded they are less significant to getting a PhD than the intelligence that can be measured by the SAT?
jsprogrammer has, I think, a different point. Let's suppose that everyone has an equal chance of getting a PhD should they be admitted into graduate school. Let's also suppose that the selection committees believe that test scores are correlated with intelligence, and that only those with a high test score (and presumably higher intelligence) should get into graduate school.
This would end up with PhDs who all had good test scores, simply because of the selection bias. It would not reveal anything about the underlying population, because it confounds any association between intelligence and getting a PhD.
The people who get a PhD this way become members of a selection committee. They know the process worked for them, and repeat it, making this a stable system.
How do you know that PhD students now aren't selected in part because of a cultural belief which places a higher weight on test scores than is justified? How much of an effect does that bias have on the correlations you keep pointing to?
Then, we would expect that there would be little correlation between testing scores and outcomes. Especially whether they get a degree. For instance, someone who gets a lower test score will have the same outcomes as someone who gets a higher test score.
But that's not what the research suggests:
> The report, which examined 1,753 studies, found that GRE scores could help predict students' graduate grade point averages, first-year GPAs, ratings from faculty, exam scores, degree attainment and number of citations earned. The specific subject tests were even stronger indicators of students' performance.
The hypothetical was meant to show that your analysis was insufficiently sensitive, not to posit that the hypothetical was actually the case.
You have an odd habit of pointing to secondary and preprint material, rather than the primary literature. Your APA links to several papers, including http://www.drtomascp.com/uploads/HungryMind_PPS_2011.pdf , titled "The Hungry Mind: Intellectual Curiosity Is the Third Pillar of Academic Performance". It concludes:
> The current study suggests that traditional sets of predictors of academic performance, notably general intelligence and Conscientiousness, should be accompanied by a third factor: intellectual curiosity. Jensen (1998) stated that “[general intelligence] g acts only as a threshold variable that specifies the essential minimum ability required for different kinds of achievement. Other, non-g special abilities and talents, along with certain personality factors ..., are also critical determinants of educational and vocational success” (p. 544–545). A remarkable number of studies on determinants of academic achievement have focused exclusively on ability and effort; the present findings, however, recommend further expanding the “g-nexus” for a better understanding of individual differences in academic performance. The latter requires—beyond intelligence and effort—a hungry mind.
It does say "intelligence is the single most powerful predictor of academic performance", then said:
> intelligence sustained the strongest effect on academic performance with a path weight of .35. TIE and Conscientiousness had slightly lower, identical path parameters of .20.
I believe that means that the other two legs together are more important than intelligence alone.
Minor issue: why does that figures' caption lead with a quip about the top 1%, but only present data for quartiles?
I don't get why there is a reference to the top 1% in the caption though.
Step 2: Sort these highly able kids into quartiles by their test scores.
Result: Even within the top 1% of IQ, the relative ability still matters. I.e. there is no "cutoff" for becoming a great scientist or inventor; smarter is always better.
If Matthews is correct that scores are transient, then we'd expect test scores to have little to no predictive ability. In other words, we'd expect the rate of career achievement among high test takers to be approximately equal to the rate of achievement among low test takers.
But this claim is directly contradicted by actual evidence:
> Follow-up surveys—at ages 18, 23, 33 and 48—backed up his hunch. A 2013 analysis found a correlation between the number of patents and peer-refereed publications that people had produced and their earlier scores on SATs and spatial-ability tests. The SAT tests jointly accounted for about 11% of the variance; spatial ability accounted for an additional 7.6%.