News for hackers. Stuff that matters.
Re:MIT to no longer consider SAT subject tests in admissions... by thebradbain on Friday March 20, 2020 @05:17PM (#22641747)
The tests don't not work (I believe that within the middle of bell-curve they accomplish their goal well), but at the extremes they are prone to gamification, especially to those in the know. For example, the pre-2017 SAT had some well known (and some lesser known) tricks that you would only know by studying the test, rather than the material:

- ALL sections (and sub-sections) have questions that strictly increase in difficulty / projected "miss-rate" as time goes on. This is to keep test takers from coming back to answers they're unsure about but may themselves know how to solve -- so if you find yourself struggling with questions in a row, it's better to stop and go back rather than miss out on what you may already know trying to solve questions that you don't. For the reading section, the scale is scoped to each passage. For the vocab section, where there are 3 sections (vocab, grammar, and multiple-choice fill in the blank), the scale is scoped to each subsection. For the math section, it is scoped to the whole thing.

- The "Free Section" (e.g. the one that doesn't count toward your score, which instructors tell you before you start that section, so you can use it as a break if you wish) is usually section 4 or 5 of the test, to help plan your breaks. Some students, not previously-knowing or confused that the "free section" is ungraded, still take it thinking there must be a penalty of some sort.

- The word "equivocal" is tested within the SAT Vocab in around 60% of tests. Unequivocally, these questions have some of the highest wrong-rates of any question on the test.

- Within the grammar questions, Choice (e) "None of the above" is 99% of the time NEVER the answer. This is one of the most certain things on the test.

- The math questions will usually have (1) answer that is an outlier. 95% of the time, this is not the correct answer; (2) will be similar to the correct answer in different ways; and (1) will be the correct answer (e.g., say you're supposed to subtract "x" by 5 to get to the real answer. The obviously fake one might be multiplied by 5. One of the slightly-wrong answers might have 5 added rather than subtracted, another might just be off by 1). If you're ever in doubt, you can drastically increase your chances at guessing on a question by picking the question "most similar" to all of the others -- something like 65% chance, rather than 25% in the naive case.

- Again for math questions -- particularly the "word riddle" type ones -- the SAT will generally purposely pick questions that could have multiple seemingly-correct questions if you plug in 1, 2, 5, or 10 for the variables. 3 is almost always a safe bet, though I particularly liked to choose 7, because who thinks you'd ever choose to plug in 7.

- The essay is funny. Per the SAT's own published rules, they are not graded on fact at all; purely rhetoric, vocabulary choice, and clarity. All of the prompts also usually include a historical figure or event of some sort -- you don't need to know anything about them other than what the prompt tells you, but a well-known and easy way to win points with the graders is to make up a fake quote from someone adjacent to the event / historical figure as a hook: e.g. "Disconsolate upon hearing the tragedy of [EVENT X], [FIGURE Y]'s au pair journaled 'His life was short, but his memory will last forever'. Previously unknown to historians until then, Y's au pair embodied Y's belief that [SOMETHING FROM THE PROMPT]. [then THESIS STATEMENT on 3rd or 4th sentence, always]." (this is an objectively wrong and terrible sentence that I would hate to read in any other context. This is, however, similar to the SAT's example of a top-tier intro).

This is just the tip of the iceberg, too. It's a very predictable format and pattern (it has to be, as it's given multiple times during the same academic year; tests must be similar, lest one session of test takers do statistically significantly better than an equally-talented group which takes the test a month later)

So yes, while I believe the SAT does attempt to test for knowledge, it's that same pursuit of a bell curve that makes it easily gamifiable for those who know the test and not the material -- who are, once again, usually already the wealthy and connected.

Re:MIT to no longer consider SAT subject tests in admissions... by knzhou on Friday March 20, 2020 @05:38PM (#22641924)
Wow, this is a remarkable level of gaming that I never was aware of!

> - Within the grammar questions, Choice (e) "None of the above" is 99% of the time NEVER the answer.

> - The math questions will usually have (1) answer that is an outlier. 95% of the time, this is not the correct answer; (2) will be similar to the correct answer in different ways; and (1) will be the correct answer

This is exactly the reason that, when I design tests, I strive to make "none of the above" or the outlier answer the correct one about 20% of the time. I really hope the SAT writers are doing that now.

> ALL sections (and sub-sections) have questions that strictly increase in difficulty / projected "miss-rate" as time goes on.

The computerized GRE goes even further... it's like a videogame, it feeds you harder questions the better you do, then reverts to easier ones when you mess up.

Re:MIT to no longer consider SAT subject tests in admissions... by fyz on Friday March 20, 2020 @05:51PM (#22642014)
> The computerized GRE goes even further... it's like a videogame, it feeds you harder questions the better you do, then reverts to easier ones when you mess up.

Isn't that one of the better ways to finely calibrate a score? Rough approximation of heapsort?

Re:MIT to no longer consider SAT subject tests in admissions... by knzhou on Friday March 20, 2020 @05:58PM (#22642069)
Yes, it definitely is!

It does feel a little crazy, though, when suddenly all the vocab words are like, "tergiversate" and "pulchritudinous". And then when you get them all wrong it's back to "the cat sat on the mat". :P

Re:MIT to no longer consider SAT subject tests in admissions... by thebradbain on Friday March 20, 2020 @05:43PM (#22641957)
I completely agree, and as much as I hate CollegeBoard hope they've done more to reinvent their tests, as I do think objective measurement of some kind does have a place in admissions, even though perhaps as not the end-all-be-all it used to be.

I used to be semi-adjacent to that sphere, but I do not know anything about the current format of the tests other than what is published.

Re:MIT to no longer consider SAT subject tests in admissions... by walshemj on Friday March 20, 2020 @06:21PM (#22642274)
Yes you need to learn to look for the distractors and rule them out
Re:MIT to no longer consider SAT subject tests in admissions... by jrandm on Friday March 20, 2020 @06:14PM (#22642211)
> The essay is funny. Per the SAT's own published rules, they are not graded on fact at all; purely rhetoric, vocabulary choice, and clarity. All of the prompts also usually include a historical figure or event of some sort -- you don't need to know anything about them other than what the prompt tells you

When I realized this applied to a lot of required essays -- ie, they can be fiction or satire -- I had a lot more fun writing papers. I wouldn't exactly recommend it but can confirm it doesn't completely tank your GPA.

Re:MIT to no longer consider SAT subject tests in admissions... by godelski on Saturday March 21, 2020 @01:53AM (#22644823)
This reminds me of a Physics GRE I took. I studied all the tests and remember very clearly the test saying to only calculate to two decimal places to save time. There was a relativity question that had 3 answers that just varied on how many decimal places you carried through your calculations. That question had the answer if you kept 2. But it also had the answers for more. I'm not sure questions like this actually show that you know the material, considering to get 3 of the 5 answers you have to do the actual calculations.

What I learned about studying for tests like these was that you were studying for the test, not the material. This felt extremely different than studying for... say... an actual physics test.

Re:MIT to no longer consider SAT subject tests in admissions... by cycrutchfield on Friday March 20, 2020 @07:36PM (#22642842)
I wish more people who fall into the kneejerk "we must emphasize only objective testing" group were aware of things like these. Standardized tests are incredibly game-able, particularly at the upper tail of the distribution. Because of the nature of that distribution, that means that if you selected students only by "objective" measures like standardized test scores, you will be disproportionately selecting for students who are experts at test-taking, not experts at being good students.

The more we can reduce the overdependence on standardized tests, the better we can be at selecting for well-rounded students and not just test-gamers.

Re:MIT to no longer consider SAT subject tests in admissions... by barry-cotter on Friday March 20, 2020 @08:15PM (#22643076)
Indeed, much better to select for the kinds of people whose parents know how these things work and can ensure their child has the right letters of recommendation, internships and extracurricular activities. And don’t be terribly specific about what exactly it v is you want; that way you can change it from year to year to ensure the right kind of people get in. Wouldn’t want Harvard to look like Cal Tech or Berkekk OK way would we? That’s what happens when you have a public, objective test instead of being able to select some smart and some connected.
Re:MIT to no longer consider SAT subject tests in admissions... by cycrutchfield on Saturday March 21, 2020 @02:05AM (#22644866)
This is not at all what I said, but by all means continue to project your own insecurities.
Re:MIT to no longer consider SAT subject tests in admissions... by ummwhat on Friday March 20, 2020 @07:46PM (#22642909)
Maybe this is will be seen as radical but I think the best way to end all this gaming is for collegeboard to not release their previous tests to the public. There simply shouldn't be enough data out there to study the test rather than the material. All that should be known in advance is the topics, format (multi choice v. Open ended) and point system. I'm sure the cottage industry of test prep companies would hate it, as would those that prefer studying the test to having merits, but those groups can pound sand.
Re:MIT to no longer consider SAT subject tests in admissions... by barry-cotter on Friday March 20, 2020 @07:55PM (#22642970)
You don’t need examples of previous tests to prepare. Even if there was perfect enforcement of no recording devices, no corruption, no breaking in and stealing test papers people would talk about what the test was like, sitting to sitting. Some people retake the test multiple times. This would probably become normative in this case because the practice effect on IQ tests like the SAT is small but real. If people were really motivated you can just pay 100 people to do the test and get them to memorize parts of it, then stitch the parts together afterwards.
Re:MIT to no longer consider SAT subject tests in admissions... by knzhou on Friday March 20, 2020 @08:27PM (#22643129)
Decreased transparency increases the advantage the test prep industry has, since they'll find a way to get them anyway.
Re:MIT to no longer consider SAT subject tests in admissions... by saagarjha on Friday March 20, 2020 @09:05PM (#22643352)
The CollegeBoard is notoriously closed with their tests.
Re:MIT to no longer consider SAT subject tests in admissions... by lonelappde on Friday March 20, 2020 @05:48PM (#22641993)
The SAT is intended to test aptitude, not knowledge. It's right there in the name.