The SAT is not fair. By taking an expensive prep course, you can potentially increase your SAT score by 100 (although that's quite extreme, the average improvement attributed to SAT prep is closer to 20 points). That's a significant, but not earth-shattering improvement: it moves you from the 75th percentile to the 84th, or from the 97th percentile to the 99th.

Now let's compare that to other criteria used in the college admissions process.

It's way easier to have impressive and relevant extracurricular activities if you're rich and go to a good school. And unlike SAT prep, acquiring good extracurriculars will definitely take years of your time and money.

Well-written admissions essay? MIT requires one. But you do it at home, instead of a tightly controlled testing center. So if you're rich enough, you can have ghost-written admissions essays. Needless to say, this process can turn even a completely worthless essay into an impressive one.

Creative portfolio? Unless you're in the 99th percentile of musical talent, the difference between "I write songs that I play on an old guitar" and "an orchestra performed my composition" is money. The former is probably not even 75th percentile; the latter, probably 95+.

Alumni parents who would be likely to donate big bucks? You don't need an expensive prep course to get that, yet it can provide your application with a much bigger boost than 100 SAT points. And unlike the SAT, if you don't have alumni parents to begin with, then you'll never get this boost, no matter how much extra work you put in.

The SAT is not fair. But it's the fairest admissions criterion used by U.S. universities today.

(note: MIT still relies on the SAT; today's announcement concerns the SAT Subject Tests)

> it's the fairest admissions criterion

I have yet to see an "abolish standardized tests" type suggest a workable alternative (or even an unworkable one).

You don't need to use the SAT or ACT as a criterion. The problem if you don't is probably two-fold.

1.) You're throwing out a signal that has proven to be pretty reliable in at least establishing a floor as what students have a good chance of succeeding and

2.) As a related matter, it's a very simple, quantitative, standardized metric (with decent predictive power) that lets you bucket applicants pretty easily.

Do you maybe throw out some applicants who are really bad at standardized tests, but would otherwise thrive in an undergraduate academic setting? Probably. But the data suggests that standardized testing is a pretty good predictor of success in school (which, of course, is not necessarily the same as success in life).

Potential directions of thought: Offer a series low cost test pilot class which allows potential students to take a single test course at MIT. The professors double as recruiters, they get to see them thinking.

Truth be told: the culture of those institutions is based on extremely synthetic virtue-signalling and superficial values and is very far from being meritocratic or open to social mobility.

This might be great for rich kids, though in case you are ambitious, clever, gritty but come from not-so-privileged background (not necessarily low-class or third world - just temporary economic turmoil is enough) - you are going to hate every single second you spend there with all your heart.

So maybe this way it is a win-win in the end. The aristocracy get their titles and the plebs their opportunities for self-realisation. No harm done.

Re: the last point; MIT says they do not consider alumni relations in the admissions process (https://mitadmissions.org/blogs/entry/just-to-be-clear-we-do...).

Is that 20 point bump a comparison taking an expensive prep class to zero preparation? Or is it a comparison taking an expensive prep class to doing self-directed preparation with a $5 prep book? My guess is the former, when the latter is what's relevant for arguments about the advantages obtained through wealth.

I took tests in the 1600 era. I came from one of the best public high schools in California. I would say 80%+ of the high school had SAT prep, every single one of the people I talked to at school had SAT prep.

If you're not including self study, then maybe? But with prep and self study, everyone I knew was able to get at least a 100 point increase, I think I got a 300 point increase. Basically aced the math section through practice and study (missed 1 question).

MIT requires interviews, too. But complaining that rich people have advantages is a bit of a "no shit, sherlock."

I'm an underrepresented minority and I increased my SAT score by 200-300 from the PSAT to SAT, depending on how you want to measure the change. This was just by doing self-study with off the shelf test prep material. The classes aren't what increases your score, its the consistent structured study and ones innate potential. For some people, no amount of study is going to get them into the 90+th percentile. While others who have the capacity to do well but haven't had the best instruction over the years can cover a lot of ground with the right prep. But this is exactly what SAT is intended to measure, scholastic potential. That some people can increase scores dramatically through preparation does not indicate a failure of the test, but rather its success.

This. I’m a college freshman and took the SAT and PSAT when in high school. I’m also not a minority. While I and my parents could have afforded the expensive prep courses the students around me were taking , I decided to skip these and focus purely on solo prep. Across the PSAT and SAT, I spent a total of $50 for my prep–a $35 dollar, thick book of SAT problems, and a $15 PSAT prep book. Technically, you don’t even need to buy these books—many libraries offer the books to borrow for free.

Just going through those books and drilling incessantly on weekends and in my free time was enough to get me a 1520 on the PSAT (the max score, which helped in gaining merit scholarships, which is the reason I go to school for free right now), and a 1540 on the SAT (99th percentile). I also used Khan Academy extensively, which provides free prep sponsored by Collegeboard.

In my opinion, prep courses are just a means for unmotivated students to put in the same amount of work as a highly diligent, self-studying student. There’s plenty of free and relatively inexpensive resort resources out there nowadays, it just requires time and attention (which sadly a lot of my generation doesn’t have).

No one even told us at my school that the PSAT would count for national merit scholarships, which ended up saving me $20,000 on university tuition over four years.

>But this is exactly what SAT is intended to measure, scholastic potential.

But if you define the variable like this, you end up opening a can of worms.

Do rich kids have more scholastic potential because they have better access to resources? Or should scholastic potential be the measurement of how a child performs assuming they had the same access to resources? Should children who belong to a racial/ethnic group that have social stereotypes of having more scholastic potential be defined as having more scholastic potential because of the effect of stereotypes and labels, or should the measure of scholastic potential be of the underlying potential if the effects of stereotypes and labels were equal?

Normally measurements of potential aim to be to measure the factors least impacted by other factors, because if we account for those factors we are measuring actual instead of potential.

> Do rich kids have more scholastic potential because they have better access to resources?

Yes. They do. It's maybe not the answer you were looking for, but if you are better prepared, you'll do better at the task at hand. Seems fairly straightforward.

The next question would be how easy are those resources to access (in the case of the SAT)? And the answer is you can do pretty well with a $20 test prep book that you can order from Amazon. The rest is time, effort, awareness etc.

Then we start getting into factors that are much harder to control or evaluate - parental involvement, guidance, nutrition, emotional support, etc.

>Yes. They do. It's maybe not the answer you were looking for, but if you are better prepared, you'll do better at the task at hand. Seems fairly straightforward.

I think you missed the point of the two questions, which is a question of what scholastic potential issue. What you are measuring is something that I might call scholastic actuality. It is how good they are with the tools they have. Potential is how good they are if we normalize the tools.

Granted, we don't have to go with that definition. If you want to call what I'm calling actuality potential, then that is fine. Just redefining a variable. But then we need a name for what I'm calling potential. Potential potential sounds a bit weird, but naming variables is hard so I'll use it as a placeholder for now.

So now that we have academic potential and academic potential potential, which should a person be ranked based off of when going to college? Given that academic potential is more dependent upon environmental factors that will change and be normalized between all freshmen (to some degree, there is still some differences), then should academic potential potential be a better predictor for success at college?

That isn't the complete answer, and real life is much more complicated than these simple equations. Just like emotional support can lead to a more mature individual that even once the emotional support is gone they have a life long benefit from.

But every metric used by colleges is to measure what you termed "scholastic actuality":

- School grades: if you didn't get the GPA you're already out of the running in many colleges

- Extracurricular Activities: if you didn't actually participate, then no one is going to look at how you might have done if you had participated

- College Essay: if you can't write you're not going to get any points here, no matter the potential you might have to eventually write something amazing

I don't see why the SAT should focus on some ethereal "potential" when everything else is centered on the current reality and not potential. Colleges aren't looking for the completely unrefined ore, they're looking for a diamond in the rough that needs to be cut and polished if you will. Not a bucket of rocks that are very likely to contain a diamond.

This is basically what I was trying to get at with my original comment - I think you did a better job of expressing it.

At some level you have to have to have something to show for your potential - otherwise there's no way to choose between candidates. Every single metric will advantage those who had the time, money, and resources to invest in developing academic aptitude. There are a small group of people who get by on developing musical or athletic aptitude but even those require time, money, energy, resources.

> Do rich kids have more scholastic potential because they have better access to resources?

Do kids with involved parents have access to more scholastic potential because their parents cared? Almost certainly, and that's a good thing and we should harvest it, not attempt to force something that is not going to work.

Once they are in college, the impact of parents is quite minified. So if you have a stupid whose default academic potential is 10 but has a parental multiplier of 2 for having involved parents and another kid whose default academic potential is 15 but has a parental multiplier of 1 for parents who don't care, once they get into college and the parental multiplier goes away the one who has a 15 should do better than the one who has a 10.

Now, numbers aren't that easily available. And maybe parents caring has a permanent effect that lasts even once they are gone. Real life is far more complicated than my example. But it shows a very simple possibility of why measuring potential while ignoring environmental failures can lead to a sub-optimal ranking.

> Once they are in college, the impact of parents is quite minified

What an absolutely ridiculous thought. Literally our entire personality, our study habits, and our grit come from our parents.

Apologies. I meant the ongoing direct impacts like forcing you to get up in time for class, enforcing a curfew, consistently encouraging time to be spent studying. The impacts on the changes to a child, are another impact, and one that is even harder to account for.

Take a simple example of waking up in time for class. You can have a college freshman who was taught by their parents to be up on time. You can also have a college freshman who always depended upon their parents to wake them up and is thus oversleeping.

The former seems more in line with the true potential, while the latter is a temporary benefit from parents. A kid who taught themselves how to wake barely on time will do worse than a kid whose parents makes sure they wake up with time to spare when parents are present, but if the parents don't instill the values of waking up on time to the latter child, then in college the former one will do better.

Perhaps, but it doesn't necessarily translate so directly.

When a kid's academic achievement is significantly caused by their parents constantly putting daily active pressure on them, the removal of that pressure can cause them to completely fall apart.

If you want a less gameable test you eventually just end up testing for IQ, but do we want to promote mere intelligence vs what you’ve done with it?

SAT prep courses advertise that they can increase scores by 100 points but generally they're lying in order to sell more test courses. But the real number is closer to "11 to 15 points on the math section and 6 to 9 points on the verbal."[1]

[1]https://www.researchgate.net/publication/228337033_Using_Lin...

As a former SAT tutor (and a guy who gamed the hell out of the SAT) I have trouble believing that. There are only a few classes of questions they ask on the test, and if you master them the whole test is a walk in the park.

Maybe the problem is most students just don't give a damn about the classes their parents make them take? Can someone explain the findings in this paper?

My impression as a former SAT instructor is a lot of students don’t spend that much time prepping. And if they have no good reading/math background it can take some effort to master things.

That said, typically saw increases well above that in my classes. Even with zero instruction I’d expect a self studying student to improve more than that from familiarization.

It did say upper class people improve more.

The SAT is designed to be an intelligence test, not a course final, and it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing.

Armchair speculation: there are probably some really bright kids out there with poor test-taking skills, and prep likely has a dramatic effect on scores for them, but most students are not that.

Did you actually sit an SAT before you started gaming? What was your improvement?

Not GP, but also a test prep teacher in a previous life. I always scored well on the SAT and test well in general, so I can't speak to personal improvement there, but a few points:

First, the test prep companies heavily encourage (and may even require?) students to take the test multiple times. That change alone will boost most people's top score by a solid margin, because it's a noisy test. The first response they always gave to people complaining about lack of improvement was "take the test again, and if you still haven't improved, take the course again for free, and then take the test again".

Second, psychometric expertise is great, but the goal of the SAT is not to be impossible to train for except as a secondary thing. It's really hard to do, especially when you are such a high value optimization target and have to build a test that doesn't rely on much specific knowledge and can be quickly scored. A lot of what the SAT courses do is just teach students to make slightly more accurate guesses on multiple-choice questions where someone unfamiliar with test strategy would leave things blank. That alone tends to boost scores, and some of the other strategies are fairly clever to help avoid common mistakes.

Last, though I didn't have room to improve on the SAT, I also taught GRE classes and can speak to my improvement there. The math section is trivial to get an 800 on (it's easier than the SAT math, or at least it was when I taught 15+ years ago), but the verbal section is quite tough if you haven't studied, and in a lot of ways is a glorified vocabulary test, and the reading comprehension sections can be pretty tough as well. Before I trained to teach the courses, my verbal score was a 490 (on the real test, not practice), so I was considering not even trying to teach (there's some threshold you have to hit, maybe 700 at the time?), but the trainer encouraged all of us to try anyways because he said the content made such a difference. After just two weekends of intensive teacher training, I tested again and ended up with a 740. After teaching the course around a dozen times, I'm pretty confident I could have hit 800 without any difficulty, you basically just have to get used to the types of questions that they ask and get in the headspace of the question authors. Just one data point, but I definitely believe that this stuff is effective.

But for someone who did well on the SAT, you would be expected to do well on the GRE too. That you were able to prep your way into a good score doesn't indicate the GRE is measuring the wrong thing. You already previously showed a strong aptitude for g-loaded tests. What you needed was familiarity with the content and the kinds of questions asked. But that doesn't invalidate the test.

> it's possible that the large team of education and psychometrics people who've been working on it for decades know what they're doing

Don't assume that society is optimizing for what's socially optimal.

Back when it was on a 1600 point scale, I got a 1300 on the first practice test for my SAT course. When I took the test for real, I got a 1570.

I wouldn't assume it, but it's one of the plausible explanations for the test appearing to meet its design goals.

Students naturally do better without test prep, too. They get older, their brains grow more, and they’re more intelligent. Much of the score increase people see is attributable to that. And most kids bright enough to exercise good test taking skills will have them without special training, I guess? 11 years of school will do that.

Not only that, but these prep courses tend to assign practice SATs that are harder in the beginning and easier at the end, to give the impression of rapid progress.

Those estimates are generally in line with what other independent investigators find, maybe a bit smaller.

The elephant in the room in these kinds of discussions is the validity of the tests to begin with. MIT makes a statement that the general tests are predictive in combination with other criteria, but how predictive?

Generally studies of these sorts of things find that they're moderately predictive of first-year GPA (like .4 correlation), and then trail off to zero as the interval between testing and outcomes increases. The effects are even smaller in studies in relatively unselected samples (due to court orders or legal decisions, for example).

So one argument always goes that you should use whatever empirically-supported stuff you can to make fair decisions, but we treat them as if they're more than they are. Sure, you could have a lot of things that are significantly predictive but with small effect size, or where there's lots of noise, but why as a society to we pretend these are huge effects?

The other thing about that paper is the hint that the practice effects are larger in higher-performing examinees, which also makes sense and is consistent with other studies. Why is this a problem? Because those are exactly the types of students for whom these issues are more applicable. A 12 point average gain with practice doesn't matter if you're including people who never had a chance at MIT anyway, but a 20 point gain might in a highly competitive group where small differences are being magnified tenfold.

The issue in all of this isn't the people in the 99th percentile versus the 50th percentile, which is the bulk of what's going into these predictive models and effect sizes, it's the 99th versus the 95th. There's a ton of real-world noise, but society acts as if the noise is nonexistent. It's like we're idolizing the outcome of some kind of survivorship bias process.

>Students were able to indicate if they had prepared through the use of school courses, commercial courses, tutoring, or a variety of preparatory materials. In what follows, a coached student is defined as one that reported participating in a commercial preparatory course.

Is there any indication that they actually broke down any further, or is this just limited to the impact of average coaching and thus not a good representation of those who receive the top tier coaching?

Note also that the measurement error of a section score is 30 points, which dwarfs the effect of prep.

The real scandal that I can't believe hasn't gotten more attention is that wealthy families are hiring psychologists to claim their child has a learning disability. They can then get more time to take the test and due to a policy change a few years ago, this information is hidden from the schools when students report their scores.

Look, faking a disability is deplorable but this stance is bonkers. I wish my parents had the money when I was growing up to get me checked and diagnose my ADHD.

And of course you wouldn't want to report to the college if a person has a disability, are you trying to scarlet letter everyone who needs a little more time to read from dyslexia?

Which illnesses count? Why does dyslexia count but not lower IQ? Why is there a time limit at all?

The point is that not everyone gets diagnosed and has equal access to performance enhancing drugs and extra time when taking a competitive test.

> What illnesses count?

The ACT has a huge list: http://www.act.org/content/act/en/products-and-services/the-...

> but not lower IQ

It's always great when people compare learning disabilities to being dumb.

> performance enhancing drugs

I realize that it's easy to look at how stimulants affect people without ADHD and conclude that they're getting the same advantage but this really isn't true. A non-ADHD med example is caffeine. Most people who down a few coffees or energy drinks will be wide awake and jittery. But for me I'll be hyper and bouncing off the walls and spacing out until I have an energy drink which lets me focus a little bit and makes me sleepy.

As someone who has used similar accommodations (though not specifically on the SAT), yes. Dyslexia will hold you back in college. SAT is intended to measure odds of success in college.

You're not wrong but I think that's due to how much of university is based around timed tests. The artificial difficulty is maddening. I was fortunate that my school's disability office had fangs and that the maths program wasn't super competitive and wanted students to learn more than get marks.

My favorite professor was teaching number theory and always scheduled his classes for the end day and gave unlimited time for exams. Every time I would sit there for at least 5 hours (which wasn't super uncommon) and it was the nicest thing to not be freaking out and trying to force myself to focus.

I disagree that it's artificial. There's only so much time in a day. A programmer who can do a task in one hour will in the long run outsprint one who needs two.

Yes, you can (like many, many others) spend an extra couple hours at work every day, but it'll all come back to center eventually. Sometimes, or even much of the time, you won't even have that option. These are times when people seriously depend on you to perform.

It's not personal. I get that it can be stressful and I've been there but I don't kid myself that I'm as good as the people who can work quickly and under pressure.

Admissions committees at top universities are some of the most bleeding heart do-Goode types on the planet. There is no way disclosing a learning disability would get you Scarlett lettered. If anything, it would probably help your application.

Yes, in some circles you are almost considered foolish not to get this accommodation. https://patch.com/massachusetts/newton/30-percent-newton-nor...

Anyone can study the SAT prep books for free at a bookstore a few hours at a time daily.

If your offended by that, what about professors or HS teachers who teach certain sections of the textbook because it's going to be on the exam.

One can self-study the SAT prep book or self-study their biology book and in both cases can potentially be more knowledgeable than those who attended a formal university class or prep workshop. Most students aren't driven to do that though.

This goes to the fundamental problem, that our university system at present is really just a credentialing and signaling system.

> If your offended by that, what about professors or HS teachers who teach certain sections of the textbook because it's going to be on the exam.

Yes, this another big problem caused by (too much emphasis on) standardised testing. I learnt the most in classes that didn't do this. But people do better on tests when they do do this. Standardised tests are destroying our education system.

Surely the standardization matters?

Teaching to an aptitude test is clearly counterproductive (and ditto tricks for hacking the test--like just testing each multiple choice answer instead of solving an equation).

On the other hand. the AP classes I took in high school (admittedly, a long time ago) seemed pretty reasonable. Perhaps you could quibble with how the syllabus weighted various subjects, or the (mercifully few) classes on how structure exam answers. Overall though, I felt like I got a decent education in Spanish/History/Physics/etc from those courses.

I think it's the testing that's the problem, more so than the standardization (although standardization can definitely be taken too far to the point that it's unhelpful). You can of course have a standardized curriculum without any testing at all. Countries like Finland put far less emphasis on test results, and a lot more emphasis on teacher's assessment of their pupils. It seems like a much healthier system to me.

I'd buy that, especially when the tests are optimized for "objective" automatic grading rather than written responses.

Diversity in knowledge across the working population would seem preferable to homogeneity to me.

Sure you want some core things, like all engineers being able to solve 2nd order DEs, but outside that you want diversity if you're looking for innovation and application to diverse fields.

In general, sure, but how much diversity you can reasonably expect for say, high school calculus? It’s not as though schools abandoned their infinitesimal-based curricula to match the AP syllabus better.

The AP US History curriculum also kind of anticipates your critique: “As has been the case for all prior versions of the AP U.S. History course, this AP U.S. History course framework includes a minimal number of individual names: the founders, several presidents and party leaders, and other individuals who are almost universally taught in college-level U.S. history courses. As history teachers know well, the material in this framework cannot be taught without careful attention to the individuals, events, and documents of American history; however, to ensure teachers have flexibility to teach specific content that is valued locally and individually, the course avoids prescribing details that would require all teachers to teach the same historical examples. Each teacher is responsible for selecting specific individuals, events, and documents for student investigation of the material in the course framework.”

https://apstudents.collegeboard.org/ap/pdf/ap-us-history-cou... (Page 11/15)

I would argue that it’s possible to have a good yet flexible standardized system; it’s just hard.

Goodhart's law in action!

The question shouldn't be is the SAT unfairly biased towards families with money, since of course it is, but is the SAT less biased towards families with money then other criteria. Having parents with money and connections lets you get tutoring to increase your GPA, tutoring to write essays, connections to get internships, the free time to do clubs, training to excel in sports, and so many other things.

Right! I don't understand why people in this thread are apparently mad rich families for this? Like a family with means is doing all the things to raise a socialized, well rounded, experienced, well-read and tutored, worldly person. Like god damn I can't think of a better use for the money as a parent.

I get that college admissions is a contest and performative but hot damn this thread is full of people mad at other people's happiness.

People are mad, because that happiness comes at their expense. Not because rich people prep their kids per se, but because society is a zero-sum game in many ways, and the end result is a de facto caste system. People are often willing to endure unfair hardships, if their children get a better chance. But if they know that not even their great-grandchildren are likely to be significantly better off, the focus shifts back to here and now.

> socialized, well rounded, experienced, well-read and tutored, worldly person

Test prep doesn't help with any of that.

The question isn't "can the SATs be gamed?", though. The question is "can the SATs be gamed more than whatever other criteria you use?". Simply analyzing one half of the equation won't produce a correct answer.

Rich people can game standardized tests. They can also game grades. They can game admission essays. They can game interviews. If colleges admitted students based on brain fMRI data, I wouldn't be at all surprised to find out that rich people managed to exploit that, too.

Are the SATs unfair? Absolutely. Are they more unfair than the alternative? I'm not convinced. I usually prefer the devil I know.

100 points isn't that much. The average score is 1000-1100. The elite universities expect 1500. Tutoring, even if it gave you 100 points, isn't going to cross that gap.

I'd say it's still relatively fair. Fairer than essentially any other admissions criterion: grades aren't comparable across schools, extra-curriculars are much more about money spent, etc.

I've worked as an SAT prep tutor and I've worked in the oil industry.

Actually, I felt way more guilt about my SAT tutoring work than the oil industry. The world runs on oil, but no one needs expensive SAT tutoring to give rich kids an edge over the poor ones.

This seems like such a weird stance since pretty much all formal education isn't free. Even without SAT prep rich families can still afford better schools, private tutors, set up apprenticeships.

It's a weird world where people are mad that those with means are using them to educate themselves and their children.

As an adult I choose to pay for lots of different forms of education that gives me "advantage" -- bleh -- over my friends, peers, coworkers, etc.. Why should I feel guilty that I have the means to buy culinary textbooks so I can bake better?

I'm not angry at the people who want to help their kids, but at the system that we're making. Our "meritocracy" has become a caste system in which a small group of people are doing very well and everyone else is scrambling to make it. It's a big social problem. Look at how suicide rates in middle America has skyrocketed, look at how much more divisive our politics have become.

I have no idea what the solution is, and can't say for sure where we should place the blame, but I have an innate distaste for the passing along of inherited advantages.

I think you're reading the position a little wrongly.

People are upset when being rich means you can be prepared for a test and so get a higher score than someone who is equally able.

Supposedly, in a properly operating university, once at the uni the background of the students shouldn't matter [as much], those who came from schools where they couldn't afford equipment should now have equal access as those who could buy anything they needed and more. Simply having access to facilities makes a massive difference to what can be achieved and that feeds in to intellect growth.

I think the difference is that, better schools and private tutors, are at least in theory, teaching useful skills and ultimately creating a better person at the end. Being good at the SATs doesn't have the same excuse, so its much more obvious when you're basically just paying to get a better score.

Look, I don't disagree with you -- other than the score SAT prep is fantastically bad education investment. But I think this is a problem with the testing more than some larger class issue. We've rewarded learning a bunch of useless facts and test taking strategies. You could probably switch out the SAT for an entire test of movie trivia and you'd probably get the same distribution of performance since ACT/SAT just reward the people who study for it the most. You can't accidentally prepare for these tests by other schooling that's not geared for it.

While it's true that prep courses can help dramatically, the SAT subject tests are also not really that hard. I don't consider myself terribly smart (certainly not MIT level) and I managed to get an 800 on the Math II and 800 on Chemistry with a minimum of studying using pretty inexpensive books. This applied to many of my friends as well.

This is one of the reasons the subject tests are not especially useful. And since they test knowledge rather than aptitude, making them more difficult doesn't actually find smarter people but instead people who had access to more rigorous high school coursework.

At the GRE level this is even more extreme. For competitive programs a perfect score doesn't distinguish you so taking the test can only harm you. When applying to grad school I was explicitly advised not to take the GRE subject tests.

The subject tests, IMO, are terrible. I took the Chemistry one right after taking AP (college level) Chemistry, and while I got a full score on that, I got absolutely destroyed on the subject test. They asked trivia-like questions and barely tested for actual chemistry knowledge.

Interestingly, I had the opposite experience, finding the AP Chemistry exam more difficult than the subject exam. I think part of it is that I have no love for chemistry, so I just memorized the things I needed for the subject test.

I was a child of the 80s, and while I agree the Math "Level 2" was a cakewalk, getting an 800 on Chem was definitely something to write home about. I get that scaling may have changed in the past thirty years, but still!

These advantages are more subtle than even that.

I was shamed by some older fellow alums when I told people that my parents had hired someone to go over my application and essays. Their stance was that my parents gave me an unfair advantage (note, both of my parents are immigrants who did not even finish high school). My retort to them was point out that most of them didn't exactly grow up in the crime ridden rural parts of the world (quite the opposite in fact). Their parents paid a higher tax rate to support the world class public education they got, which they just assumed is the norm. That shut them up rather quickly.

SAT prep classes or not, coming from a more advantageous social-economic background grants you so many benefits in many ways. My parents wouldn't have to hire someone to review my applications if my school had a good college counselor (50% of my classmates do not go on to college).

>coming from a more advantageous social-economic background grants you so many benefits in many ways //

It's not all benefit: you can be taught to be lazy, not to be self-reliant, you can lack tenacity and resilience, you can have perverse expectations and a sheltered view of the World. Such things can turn out to be a hindrance to self-fulfillment.

Yeah that's a fair point for sure and I've seen examples of those too. However, even then some well resourced parents will still clear a path for these children to get into selective schools as the recent college admissions scandals revealed.

The main thing that frustrated me about the SAT, at least when I took it, was that it essentially boiled down to an IQ test, not a true "scholastic aptitude" test. I did barely any prep (aside from, I think, a flip calendar with questions), but I scored 1510 out of 1600 at the time which qualified me for a prestigious program in my university.

Then I narrowly avoided flunking out because I'd never actually learned to study or manage my time properly in high school because I'd never needed to until college. Study skills and time management seem like pretty important factors in a true measure of scholastic aptitude.

Similar story to you except I graduated with a 3.4 from a top 10 university without learning to study or manage my time. I guess the test was right?

This reads more like a humblebrag to me than a thoughtful criticism against standardized testing.

That's a fair criticism of the general SAT - I used to teach those classes, and the test is very gameable in the sense that there really are "A FEW SIMPLE TRICKS!" to learn that have nothing to do with actually knowing the material in a useful way.

But at least some of the subject SATs are not like that at all, specifically the math/science ones. There really aren't many tricks or traps, they really are like normal school tests (if multiple choice) where doing well on them requires you to know the material they cover. Nobody who is "good at tests" is going to 800 the physics one without knowing physics well enough that they'd do well in a freshman mechanics course, and someone who gets a 400 either slept through class or is going to struggle at college level.

Coming from someone who took said expensive prep courses, I can tell you that just taking practice exam after practice exam is easily 90% of the benefit. The test taking strategy they teach is something anyone with serious prospects of getting into MIT would have learned years ago.

What really increases your score is having a good night's sleep and a healthy breakfast before school every day of your life prior to taking the test. The test doesn't really measure aptitude because it can't control for those inputs.

Those inputs cause aptitude.

You can get like 80% of the value of an expensive sat prep course from a cheap sat book.

But money also buys you grades, gpa, extracurriculars, and recommendations.

I work for a company that collects aggregated SAT data. I was pretty upset when I discovered there is a very strong, direct correlation between family income and SAT score, with a credible spread of several hundred points between the average scores of the highest and lowest income groups. I knew that there would be a correlation, but seeing how strong it was was pretty depressing.

Just so we're clear: IQ does correlate with income, for obvious reasons.

Why is that depressing?

I think the courses mostly help with discipline in studying for the test. There’s nothing magical about them. I’ve taken many standardized exams with and without prep courses and it’s quite possible to do very well with inexpensive self study material if you’re incredibly disciplined about studying.

That's true, but at the same time it's a standardized test so anyone can replicate those prep courses. There's tons of prep books with practice questions, tips, etc.

It's a matter of awareness and taking it seriously (obviously a simplification, but hope you get my point).

I mean, it's not really true. People spend a lot of money on test prep, but this isn't even a case where the evidence hasn't come in yet. We know perfectly well that test prep has negligible influence on SAT scores.

Those studies were mostly funded and influenced by the college board, and they needed that to be true, so that’s what they found. But there is prep and there is prep!, they are different with different influences. Studying the former (say a few weekend long prep course) to make conclusions about the latter (intensive cram school for a couple of years) is criminal.

I’ve seen an SAT cram school in China, and yes...it will influence your scores significantly because they pull out all of the stops.

> Those studies were mostly funded and influenced by the college board, and they needed that to be true, so that’s what they found.

The SAT draws a huge amount of overtly hostile attention. So does SAT prep. But these highly motivated SAT-hostile researchers haven't been able to produce the results they need.

That simply isn’t true, eg see https://www.washingtonpost.com/news/answer-sheet/wp/2017/05/...

The College Board can’t really have it both ways. And of course: https://www.reuters.com/investigates/special-report/college-...

Which is just one example of pulling out all the stops.

Yes, no one disputes that memorizing an answer key can give you a higher score. But by the same token, no one believes that the influence of cheating is of interest when studying potential gains from "test prep".

That isn’t just it though: they have access to old tests, they have access to test takers who memorize questions, given the importance and popularity of any test, these options will arise and people with means will have access to them.

There is no avoiding Goodhart’s law.

You can make a case for a much weaker form of Goodhart's Law, that with the SAT as a well-known target, SAT scores are less informative than they would be otherwise.

But "there is no avoiding Goodhart’s law" is obviously wrong if you interpret Goodhart's Law as it's actually phrased; the SAT is doing a fine job of avoiding it today. The SAT hasn't ceased to be a good measure. It's still very informative.

>And those expensive prep courses had the potential to increase scores over 100 points.

I took one of those expensive scores and got the exact same score after the course as I did before the course.

It turned out the cheap study book was every bit as effective as the expensive course.

I realized how much of a scam standardized testing was when I used to do the PSATs and increased my score mostly by using a different strategy rather than getting smarter or anything. The SAT prep class my high school offered for like $40 was a complete joke because they were designed for public school students where you get around 1250 and it's pretty good for state school admissions. Until I learned to treat the test as a game and understand its scoring I went from 1330 to 1520 simply by _not_ answering anything I wasn't fully confident with my answer - this is literally the opposite strategy I was instructed to do where they encouraged people to guess. To get to the higher end, you are best off never getting even a partial deduction for getting something wrong. When I got my best score, I only got 3 questions wrong but far more unanswered.

> And those expensive prep courses had the potential to increase scores over 100 points.

As compared with self-studying with free online practice tests and resources having the potential to increase scores over 100 points.

How is that functionally different from most things considered by college admissions? Money helps a ton for normal class, extra-curriculars, etc.

If you gave me a student who had enough motivation to succeed, I bet I could raise his score far far far more than just 100.