Notably it’s not clear if there’s a better determinant of that. Tests at least have the advantage of being a knowable thing you can prep for, instead of whatever you have without tests.

    > IMO the thing that tests do not sample is persistance

See Angela Duckworth's research on "grit"; it is indeed very difficult to test for.

No, it's not hard to test for. Military units test for it routinely. At the SEAL/Delta/Army Special Forces level, the testing for that is insane.

Some companies used to do this. Around 1900, if you went to work for a major US steel company, you started out shoveling coal. Even if you were destined for management. Japan Air Lines pilot training for new pilots involves a stage where they do grunt jobs in maintenance such as checking incoming part serial numbers.

    > No, it's not hard to test for. 

Hmm...

    > ...the testing for that is insane.

Sounds like it's...hard to test for.

I would agree it is hard but not impossible to test for. The test being very long, at some point people just figure no one can finish it... and in some ways that is how I feel about work in life.

I imagine if you told people "we are not testing for correctness, just number of problems solved" then you would skew this test.

However when the tests were administered in the longitudinal study, the researchers didn't know which metrics would be important so they just put a lot of problems on it.

Disagree. You can retake the tests, hence they do test for persistence.

this clearly pales in comparison to the extended effort for 4 years of highschool that is factored into your gpa

In most cases, I'd agree with you, but my parents thought it would be a great idea to have me start taking the exams in 7th grade so that by the time it mattered, boy would I be ready.

Over four years, I added 350 SAT (1200-1550) points and 5 ACT points (30-35), but it seems pretty likely to me that it wasn't the direct work ("persistence") that made the difference as much as the generic four more years of schooling and general maturity.

I wouldn't say you "added 350 SAT points" so much as your score went up by 350 points between 7th and 11th grade. I would imagine that for practically every student, SAT scores would go up by hundreds of points during this stage of academic development.

> but it seems pretty likely to me that it wasn't the direct work ("persistence") that made the difference as much as the generic four more years of schooling and general maturity.

Retaking tests, to optimize a specific metric, I think is a direct misunderstanding of this point.

If one sets a goal to achieve & takes many tries at it, it is similar but not the same as "heres a bunch of questions, answer as many as possible and some correlations in the answers might be more important than others".

I think if you want to predict success there are two key factors: 1. Test takers must have weak feedback on their results 2. Test takers must be told that there is not uniformity in the importance of specific questions & some may be worth more than others. The test must be long enough that no one is willing to sit for the entire duration, and test takers should not be given an end-time or instruction on how to take the test.

Why? #1 mimics the weak feedback in real-life scenarios, if someone doesn't find you likeable for some unknown reason ... you'll probably see less of them.

#2 gets at you may never really know whats key to your success & hopefully by trying you get "lucky".

I didn't "train for the test." There were no SAT books or training courses or anything like that.

Sure, taking it a few times meant I knew what it felt like to take it. I knew what the books were like and how the proctors would act. I knew what could go wrong and what kinds of distractions I should be ready for.

That's why I think I only actually got better at the same rate that I would have if I weren't taking the tests.

Understood. I think that if one wants to predict success / happiness later in life the test should be as much like real-life experiences as possible.

A carefully crafted test-taking strategy is an optimization that seeks to maximize the score. Scoring on the aforementioned test basically wasn't correlated with future outcomes in a significant way, ie it could contribute but isn't a major, main or significant factor in determining how happy or successful people who took it came out.

Retaking tests has these problems, in that people adapt to optimize the result. For example if you told people the result of this test before taking it, I would say you would end up with a lot more people "answering" in the fastest way possible all of the questions. This would destroy the effect measured, in the same way that telling people tests will be ranked by scores erodes the metric.

it would only measure persistence in test-takers who score low enough that they want to retake it.

The test is too long to finish, basically at some point the students look through the pamphlet and decide no one can finish this thing.

"Retaking" misses the implementation of this test, it is more akin to "I'm going to keep giving you problems as long as you are willing to sit here" kind of thing.

Additionally this is the main correlation of the study. I doubt if researchers had in mind that the number of questions answered would be significant. If you told students "we are testing for number of questions answered" obv would distort the results.

Wouldn't someone with grit will find a way to persevere and thrive regardless of which institution accepts them?

Probably, but you want to give the network effects to them to multiply their societal effectiveness. Long term rewarding them is less about assisting them individually but increasing their influence on the general welfare.