Is there a correlation between SMPY eligibility and getting into elite universities? Probably, but it doesn’t seem like an improper one. Is there a correlation between elite universities and life outcome? Sure.
Does that make the longitudinal study from age 11/12 into adulthood useless or content-free? Not in my opinion.
Only Study 2 was done on cohort 5, which was selected as grad students already at elite universities.
I also realized that SAT scores are heavily impacted by SAT specific training/tutoring that costs a reasonable amount (because as far as I can tell - as a foreigner - the SAT tests rely more on having been taught the exact way - and tricks - to answer SAT questions in the time available. Honestly if your test is good it should not have any real time pressure for the vast majority of participants)
"From 1981 to 1990, three separate analyses of all the prior studies were published in peer-reviewed journals. They found a coaching effect of 9 to 25 points on the SAT Verbal and of 15 to 25 points on the SAT Math. In 2004, Derek Briggs, using the National Education Longitudinal Study of 1988, found effects of 3 to 20 points for the SAT Verbal and 10 to 28 points for the SAT Math. Donald Powers and Donald Rock, using a nationally representative sample of students who took the SAT after its revisions in the mid-1990s, found an average coaching effect of 6 to 12 points on the SAT Verbal and 13 to 18 points on the SAT Math. Many studies tell nearly identical stories. On average, coaching raises scores by no more than a few dozen points, enough to sway college admissions in exceedingly few cases."
The extent of my tutoring was a public school guidance counselor explaining to our whole class who was taking the test that there was statistically no penalty for guessing and how the “A is greater, B is greater, C means equal, D means indeterminate” questions worked. That was 30 minutes at most.
High SAT scores are evidence of high SAT score.
It's difficult to realize something that isn't true. The effects of training on SAT scores have always been negligible, less than the measurement error associated with taking the test one time.
When I was in high school, I would check out whichever ones were available at the time and would photocopy the whole book. I drilled myself on the tests until I scored 1600s every time. Shows how flawed they are as a testing vehicle.
That is evidence in favor of the test's quality, not against it. There's no paying to win.
There is no reason to control for those, and doing so would be an 'Everest regression': "controlling for altitude, Mount Everest is actually warm today". What you are asking for is like reading about IQ and income and saying "ah, but did they control for education?" Controlling is not magic pixie dust.
that thing that literally raises the measured IQ number?
you know that practice can raise your score, right
The point stands however: controlling for educational background is a thing you can do in such a study as according to the factors I pointed to, and gwern's glib comment is wrong
Your suggestion - with this fringe talk about "the sociologist's fallacy - is to eliminate any explanation from any study which does not assign success to invariant psychological metrics.
Then what exactly do you think you are doing when you rant about how they didn't control for it?
> Your suggestion - with this fringe talk about "the sociologist's fallacy - is to eliminate any explanation from any study which does not assign success to invariant psychological metrics.
Nothing fringe about it, I'm afraid. This is well-understood in the statistics fields like causality you seem so loathe to actually learn about.
The point is: when you control for something and then go on to make causal claims, you are not making a minor tweak. You are assuming an extremely implausible causal model and making strong inferences from there about what are real causes and what are not. You have entirely failed to argue for a causal model which would justify simply controlling away family background etc, and it seems from how your comments handwave it away in favor of nothingburgers, you don't even understand what you're arguing for.
clearly we're talking about the impact on IQ of whatever factors we make up to describe IQ, and obviously, and provenly, education (and therefore the ability to get education) is in play here, which gwern's glib rebuttal ignores
And that's not relevant anyway. Are you trying to suggest people only get into SMPY because their parents made them practice the SAT? In the 1970s and 1980s, hardly anyone would have even heard of SMPY before their kids took the test in school.
and predict results pretty well too
Nor does zip code predict remotely as well as IQ.
Not everything is about wealth.
STEM graduate school doesn't select for wealth at all. Undergraduate admissions sometimes do, but essentially zero of the wealth-based admits major in STEM. In fact in my experience the effect is the exact opposite of what you say: STEM takes in whoever can do the work, regardless of wealth, and helps produce new wealth. In every institution I've been in, STEM people had the lowest average family wealth by far.
Though this is obvious, it wasn't obvious that the tests given were actually showing who had some amounts of skill. Though, this should more accurately be called "some amount of ability to become prominent in a field," as like you say things other than skill may cause both high scores and success later.
As a thought exercise, you can ask yourself: would you prefer that your doctor be from the former or the latter group?
If anything I see selection for wealth: the samples in one of the bigger studies selected from “the 15 best STEM graduate training programmed”, which is broadly a proxy for wealth.
More over they don’t have negative samples: afaict their “study” was to take the top X% of students from some already wealth biased institutes and compare their performance to the overall average for everyone. What they should have done is compare the performance of the X% of the class they sampled from with the performance of the remainder of that class.
Then we could compare more similar starting points.
Obviously we would generally expect some correlation between skill when younger to 35 years later, but I would expect the entire group to do significantly better than the general average. Eg a person who got into Harvard because of their family history, and gets a C, is likely to do better than a person with a 4.0 from a state university. (General averages, obviously there can be outliers in every direction)