This is the detail which needs to be added to almost all mainstream coverage of education stats — every year you'll see some hand-wringing about U.S. students falling behind internationally and the authors never mention that the effect is much weaker once you correct for the fact the U.S. has a much wider range of students taking most tests. Usually almost all of these metrics come back to the root cause of economic disparities along the spectrum from hard poverty to not being able to afford tutors and SAT prep courses.

(This is also why the cool curriculum ideas or charter school concepts we hear about never go anywhere: the TED Talk sounds great but inevitably once it scales up it turns out that the effects were due to some sort of selection bias (quite possibly unintentional) or simply lucky small sample sizes which peters out at larger scale)

> the U.S. has a much wider range of students taking most tests.

Citation? All countries have a mix of wealthy, educated students, working class students, students whose parents are drug addicts, immigrants who don't speak the local language, etc. In my experience the US isn't that different from other wealthy countries I've lived in.

SAT isn't really used for international comparison; a system like PISA does cross-sectional studies of countries, not elites. The results are not encouraging for the USA:

https://en.wikipedia.org/wiki/Programme_for_International_St...

https://nces.ed.gov/surveys/pisa/

http://www.oecd.org/pisa/

> Citation? All countries have a mix of wealthy, educated students, working class students, students whose parents are drug addicts, immigrants who don't speak the local language, etc. In my experience the US isn't that different from other wealthy countries I've lived in.

Yes, but many European countries engage in so-called tracking of students, putting them in separate tracks for tradeswork, skilled labor, and university such that the "dumb" ones aren't taking university entrance exams.

In the US this has become increasingly frowned upon by (perhaps) well-meaning administrators who feel that to do this is unconscionable. In my freshman year of high-school they had advanced academic and standard groupings in the same school, but removed this my sophmore year. They re-branded all the base-level courses as "College Prep," with "Honors" being the former advanced-track classes, AP and IB courses already being branded as such.

Of course, if you were in the advanced group the counselors and teachers would all push you into AP and IB courses but they also encouraged everyone to take the SAT and ACT tests, to apply to university, and discouraged enrolling in the trades program that the county offered (half-day academic, half-day trades classes).

This type of "everyone goes to college" mentality is why the SAT is a poor academic bellwether.

But in Europe this often goes hand-in-hand with mandatory schooling while in US people sort themselves by dropping out.

Mandatory schooling in most of Europe is up to the age of 15-16 and you can do w/e you want after that not unlike the US...

https://os.unil.cloud.switch.ch/tind-customer-edudoc/aa7b2aa...

In my home state, one cannot drop-out until 16 which is well past when one would be traditionally tracked.

>In the US this has become increasingly frowned upon by (perhaps) well-meaning administrators who feel that to do this is unconscionable.

Because it is unconscionable. I dropped out of high-school at 16. I was able to go back and earn a degree in computer science at the age of 28 at a Canadian school. Under the European system where your life path and career trajectory is decided for you at 14 years old, I wouldn't have been able to do this.

What about tracking students in high-school precludes your being able to attend university at the age of 28?

Insofar as I'm aware, one can change tracks in most tracking systems. It just isn't entirely common, likely because it puts most on a path where they should be.

> What about tracking students in high-school precludes your being able to attend university at the age of 28?

Because the "dumb" students are pushed into trades instead. And good luck going back to university when you don't have a long chain of prerequisite courses from age 14 onwards. In the states you can just get a GED and take the SAT (along with individual selection) to prove you belong in university.

Assigning a person a career based on a test they take at 14 seems awfully reductive and like it would segregate students based on the social standing of their parents. Wouldn't surprise me if the German system reduces social mobility.

Should students have their entire lives mapped out for them at 14? And if so, why at 14? Why not at 10, or 6, or at birth?

Nothing about a tracking system actively precludes your being able to jump tracks later in life. To be frank, you're blindly speculating based on this assumption.

> Assigning a person a career based on a test they take at 14 seems awfully reductive and like it would segregate students based on the social standing of their parents. Wouldn't surprise me if the German system reduces social mobility.

That's not how tracking works. They don't assign you a vocation.

>Nothing about a tracking system actively precludes your being able to jump tracks later in life

Nothing necessarily precludes it but does it really happen that often? I couldn't find any data on this. Is there any evidence that "changing careers" is easier and more likely under the German VET system as opposed to the US system? In fact, evidence suggests that the permeability between the academic and vocational tracks is quite low in actuality [0]. One would logically think that making long "on the track" prerequisite chains a requirement for a career change would add significant barriers to entry. And there is some evidence that supports my theory that VET-like systems increase social stratification and socioeconomic inequity [1][2]

>They don't assign you a vocation.

A difference without a distinction IMO. They assign you a track of schooling which will leave you qualified for only a certain subset of jobs.

[0] https://www.oecd-ilibrary.org/education/a-skills-beyond-scho... page 38

[1]: https://link.springer.com/article/10.1186/s40461-016-0033-0

[2]: https://www.semanticscholar.org/paper/The-social-stratificat...

> Nothing necessarily precludes it but does it really happen that often? I couldn't find any data on this.

Does it happen often in the alternative system? You started with saying that you left school at 14, then went to university at 28. No matter the system that's going to be an unusual situation.

You would just spend 1 year taking the requires courses in an adult education school and then getting a 3 year major. Seems pretty equivalent to doing a 4 year American major which had no prerequisites.

> Because it is unconscionable. I dropped out of high-school at 16. I was able to go back and earn a degree in computer science at the age of 28 at a Canadian school. Under the European system where your life path and career trajectory is decided for you at 14 years old, I wouldn't have been able to do this.

Would tracking you differently have perhaps resulted in you being able to graduate high school?

You can just redo the 2 years of high school you missed out on.

> Yes, but many European countries engage in so-called tracking of students, putting them in separate tracks for tradeswork, skilled labor, and university such that the "dumb" ones aren't taking university entrance exams.

Err, this is a positive thing? Personally I found the AP classes and SATs to be pretty useless beyond getting into college. We should just admit anyone who wants to to.

If the point of this is to justify high international test scores, there’s a much better argument there to test either nobody or everyone.

Community colleges admit anyone who wants to go. Students who do well there can then transfer to complete a Bachelor's degree. It's a great option for students who perform poorly on standardized tests.

This doesn’t change the fact that meritocratic admission is fundamentally broken at the other schools.

There are limited resources to provide high-end education. Effective use of those resources is key, and meritocratic admission is far preferable here to "just let everyone in".

> We should just admit anyone who wants to to.

Well, the problem was not only that anyone could enroll in the higher-track classes but teachers and counselors pushed many to take courses they weren't prepared or willing to take. I myself was heavily pressured into taking higher-level Spanish courses than I was capable of, or even willing to take. I was explicitly told I had to because I wouldn't get into any universities without it. They really wouldn't take no for an answer, and I ended my senior year with a 40 / 100 grade, but thankfully had a high enough GPA and enough credits to take the hit without issue.

In almost every class one could expect the population to be bi-modal, you would have the flunkies with no desire or hope to succeed and you would have the students who wanted to be there. This resulted in the teachers often having to meet the lowest-common-denominator.

We had a programming course (Visual Basic) that after the first month they switched to one of those "learn-to-program" visual building-block languages, but then myself and a few others raised hell so they allowed everyone to choose which they wanted to do in the same class of 30 or so. The reality was the counselors pushed the students to take the course, and most figured they would get a free 90 minutes of fucking around on the internet, and the teacher ended up giving the lower-end of students low-but-passing grades.

As it were, I'd had that same teacher several years in a row and ended up as something of his assistant because I was already a competent programmer. They're not able to do anything, it's all a metrics / numbers game.

To be clear, my point isn't about the benefits of tracking as it is a rebuke of a broken system of metrics-appeasing administrators cloaked in the lie that everyone is equally capable of the same things.

> Yes, but many European countries engage in so-called tracking of students, putting them in separate tracks for tradeswork, skilled labor, and university such that the "dumb" ones aren't taking university entrance exams.

Not at all, this is largely false.

Instead of downvoting, provide evidence to the contrary.

In various countries in Europe where I lived there are no laws preventing people from taking university entrance exams.

> Citation? All countries have a mix of wealthy, educated students, working class students, students whose parents are drug addicts, immigrants who don't speak the local language, etc. In my experience the US isn't that different from other wealthy countries I've lived in.

If you look at [0], It looks like the parent is right.

> However, the PISA’s results are drastically over simplified. For instance, as noted in a report by Dr. Martin Carnoy of Stanford and Richard Rothstein of the Economic Policy Institute, American students actually perform better than the much higher ranked Finland in algebra in general, but worse in fractions. More importantly, when you normalize the results between the countries, adjusting for the relative poverty of the students taking the PISA tests, the U.S performs significantly better, ranking 6th in reading and 13th in mathematics, a drastic jump in both categories.

> Dr. Carnoy and Rothstein further note in their report What Do International Tests Really Show About U.S. Student Performance? that when you divide the kids based on family wealth, the actual gap in performance isn’t so stark between any country, with a not insignificant portion of the ultimate ranking of each nation being based on how many impoverished vs. middle class vs. wealthy students are taking the tests.

[0] http://www.todayifoundout.com/index.php/2015/01/kids-get-sum...

> when you normalize the results between the countries, adjusting for the relative poverty of the students taking the PISA tests, the U.S performs significantly better, ranking 6th in reading and 13th in mathematics, a drastic jump in both categories.

We're doing better than that. When you break the results down by race, the US generally ranks second or first. (US whites do better than whites in Europe; US asians do better than asians in Asia, US blacks do better than blacks in the few countries with a large black population that aren't afraid to take these tests in the first place...)

Would you mind providing a citation for that? I've genuinely never heard that.

2012, reading: https://isteve.blogspot.com/2013/12/pisa-reading-scores-by-r...

2012, reading/science/math reported separately: https://isteve.blogspot.com/2013/12/overall-pisa-rankings-in...

2015, average of reading/science/math: http://www.unz.com/isteve/2015-pisa-mean-scores-in-perspecti...

This is not the stuff of an "educational crisis".

The US has a caste-like professional class and an underclass.

You can pretty easily observe the difference by looking at school performance in the x largest suburban districts in your state and comparing it to the x largest urban districts.

Wasn’t Shanghai famous for getting really good PISA scores, that being the richest city in China?

The Shanghai results are quite controversial, not only in terms of accuracy but whether they might skew Chinese results significantly.

Not being in the field (and not really caring much about it TBH) I don't know how to evaluate the controversy or even its legitimacy.

The US has far more ethnic diversity than any country that’s at a remotely similar level economically. If you compare the average scores for Americans by race to countries where that race is the majority the US looks pretty great considering it’s 1/20 of humanity, less than ten countries do better at educating White or Asian populations, one does better with Hispanics and US Blacks outscore the only majority Black country that takes part in PISA by a small amount. You see the same dynamic in the US by state.

http://www.unz.com/isteve/2015-pisa-mean-scores-in-perspecti...

>The US has far more ethnic diversity than any country that’s at a remotely similar level economically.

That sounds like a cherry picked criterion to make a point.

In any case, looking at GDP per capita, Singapore is similar to the US, and also similar in terms of racial/ethnic diversity. Yet Singapore (as a whole) tops your rankings, and the US as a whole is way down in the list.

>The same census also reports that about 74.1% of residents were of Chinese descent, 13.4% of Malay descent, 9.2% of Indian descent, and 3.3% of other (including Eurasian) descent

That's extremely different than the US.

Different in terms of the ethnicities involved, but not in terms of the distribution:

72.4% White, 16.3% Hispanic, 12.6% African American, 4.8% Asian

Honestly, I'd say that a good chunk of that is culture. Singapore has an insane focus of education, with enormous amounts of spending on supplementary education and such.

In the US, the Asian "tiger mom" is seen as an odd peculiarity. That alone speaks volumes.

>Different in terms of the ethnicities involved

That's the entire point though Africans and Hispanics score lower than Asians and Whites, as shown in his link.

Like it or not, different racial groups have significantly differential SAT mean scores. And the US has a growing number of minorities that do not do well on said tests.

> from hard poverty to not being able to afford tutors and SAT prep courses.

I went to a pretty fancy prep school with a lot of rich folks, but I don't think most did any special prep for the SAT. I think we had a class period where they talked about the sorts of questions that appear on the exam, told us about when/where it'd be offered, etc., along with ads for SAT-prep services, but I think a lot of folks ended up deciding it would be best to get a good-night's sleep before walking in on it. (We also debated stuff like if drinking coffee or/and having a chocolate bar beforehand would be helpful.)

Which isn't to downplay the advantages that affluent students enjoy, just, I don't think that SAT prep courses are really the big thing. (Plus I suspect that students who spend a lot of time doing test-specific prep are losing out on their actual education that they'll actually need later on.)

    > I went to a pretty fancy prep school with a lot of rich folks...

I suspect you might not be aware of what a vast number of public schools in the USA are actually like. SAT prep becomes more and more critical for students as they come from increasingly underperforming high schools.

There are schools where ~1/3 of the students just wander the halls aimlessly instead of going to class, where teachers turn over at a rate of 25%/year. In such schools even students who intend to go to college are going to be utterly unprepared for the SAT. These students actually need a prep course.

I'm not sure I follow your point.

I mean, hypothetically, let's say that a student never attends class, just aimlessly wandering the halls (which I'm skeptical about, but let's go with it). Then even if they somehow game the SAT and get into a good college, then what? If they're basically uneducated upon entering college, then it seems like, best-case scenario, they'll get like 4 years of education in college -- making them analogous to a student with a gradeschool education, right?

I mean, this sounds like a smoke-and-mirrors scheme.

I am not saying that the students aimlessly wandering the halls are taking the SAT-- they're in the jail or dead-end-job pipeline. They are NOT going to college.

But there are other students in such schools, however, that are trying to make the best of the situation they're in and these students want to go to college. Sadly, these school is so busy with discipline and attendance issues that it's almost impossible for serious students to get prepared for college.

And yes, there are many public schools that would appear to a visitor as chaos. Students do just roam the hallways if they don't feel like going to class. There can be so many doing this that it looks like they're changing class-- all the time. Not exaggerating.

Don't be skeptical. I know several teachers in the parts of big cities where many classes are just the teacher trying to keep the kids from hurting each other and any lectures aren't really listened to, where teachers are cussed at and assaulted with zero consequences. There are many schools where kids are effectively offered zero education because the environment is too chaotic and violent.

> just aimlessly wandering the halls (which I'm skeptical about, but let's go with it)

For some schools it is a real problem. My girlfriend works at a pretty bad high school and she has tons of students who keep failing their classes because they never show up and the principal literally wanders the halls himself to try and force kids to go back to class.

Same: in 1981 our SAT "prep" was literally two sentences one morning: 1 - there will be little bubbles you need to fill in and you must use a pencil and 2 - some statistical strategy concerning guessing. Everybody took the test only once; I don't remember any efforts to game the system.

But that in itself points out the advantage: the test was basically designed to reflect the level of instruction my school already gave. If you don't get that from your school (or believed you don't) you would have resorted to additional study and the like.

Likewise back then there weren't "AP" classes and the like: if you wanted to take the AP test you could but there were no special arrangements at my school.

I don't know how much things might have changed: my kid's SAT prep in 2017 was simply to purchase the book but not open it. On the other hand there's a whole apparatus of AP infrastructure with special "AP" classes, different scoring for those classes(!!) etc. AN enormous kabuki of infrastructure that as far as I can tell signifies nothing.

> I think we had a class period where they talked about the sorts of questions that appear on the exam, told us about when/where it'd be offered, etc., along with ads for SAT-prep services, but I think a lot of folks ended up deciding it would be best to get a good-night's sleep before walking in on it.

That's actually non-trivial amount of prep, but I personally don't know anybody that took the SAT "cold". Everyone prepped quite a bit - I prepped for months and got mediocre scores consistently ¯\_(ツ)_/¯

Yeah.. prep doesn't seem too effective. That's what they told us at prep school (heh). I mean, they basically just said that we ought to ensure that we have sharpened pencils; adequate batteries for our calculator; a good-night's sleep; and to show up early.

I can appreciate how some folks want to do better, and are willing to work for better scores by investing in test-prep. And I can understand how test-prep providers would encourage them to do so, talking up their own services, as any business would.

But, I don't think it's that big a thing in the end. If anything, I worry that students who spend a lot of time prepping are being cheated out of their time, money, and education.

---

PS: To be clear, students would definitely do well to spend time studying for their future. Just, instead of trying to game the SAT, they can learn something useful, whether that's math, physics, computer programming, philosophy, creative writing, or whatever.

> I personally don't know anybody that took the SAT "cold"

How do you define "cold"? IIRC, my prep consisted of just taking the PSAT. I ended up with a 1370 (700 verbal, 670 math), which was good enough for my purposes.

I also don't recall any particular preparation that I did for the SAT - I may have picked up a second-hand prep book somewhere.

But it was not like it was rocket science to know how one is supposed to take these kinds of tests. I graduated from a very unremarkable rural school system in New England. From 2nd grade on, we took a standardized scantron-type test (the CAT test, if I recall correctly). In 4th and 7th or 8th we took a state-wide standardized test that was more of the same multiple-choice sections, with an additional writing component that was a watered-down version of the now-discontinued writing portion of the SAT. Then I believe that they had us all take the PSAT in 10th or 11th grade, and they also encouraged people to take the ASVAB, as we had a large percentage that were planning on the military anyway.

By the point it was time to take the SATs, all of the strategies and gimmicks that are covered in test prep for the SAT were pretty old hat, and we'd been through that wringer at least a dozen times over by then.

> Usually almost all of these metrics come back to the root cause of economic disparities along the spectrum from hard poverty to not being able to afford tutors and SAT prep courses.

Do you have any resources for how much an SAT score can change from studying? In my experience offering free ACT tutoring at the local YMCA, I wasn't able to improve scores that much. Maybe I'm just a bad tutor. I did pretty well on the SAT when I took it, with little prep.

My experience with standardized testing is that there are some unprepared students who are not familiar with, or fail to use basic strategies like process of elimination, jumping between question and text, etc and these students will get a hugely significant boost, but beyond that progress from tutoring at best equal to self studying. I also did well with little prep but that was because of my highschool experience.

My experience working with outreach programs taught me that we would see a 1 or 2% increase from focused efforts on content mastery in SAT/ACT prep classes. When we focused on specifically the skills needed to do well on 'generic' standardized tests, we saw a 7-12% increase in individual student score.

These tests mostly test how well someone (a) responds to pressure and (b) takes a standardized test. They're not accurate measures of student learning.

This was my experience. In my SAT prep class 15 years ago, the first thing the instructor did was talk about the statistics of guessing and how even if you don’t know any answers at all, you can improve your score dramatically just by eliminating the obviously wrong answers before you guess. Then we went about learning ways to find the obviously wrong answers based on the format of the question that was asked. We also took practice tests, but it was the generic test taking rules that made the difference for me. I did very, very well on the SAT and then later, I applied the same rules to the LSAT and did much less prep and did very well on that too.

I appreciate your anecdote and that lines up with my experience. But a 10% increase is what I had in mind when I said I couldn't increase their scores much.

It doesn't seem to me to support the idea that the SAT tests mostly your (a) and (b). The average SAT at MIT is 1530 with the average at my local state school University of Indiana at 1250. Prep classes for the average UI student increasing their score 10% to 1370 wouldn't even get them to the 25th percentile MIT student.

It seems to me the SAT does capture something beyond simple ability to take a test.

If that were true then teaching how to take the test would have resulted in a larger jump on scores.

I have a 10th grader who is just starting out on his SAT/ACT journey. A combination of Khan Academy[1] and a $10 book is his prep. I'm interested in how much these resources and the deliberate practice that goes along with them will improve his baseline scores.

[1] https://www.khanacademy.org/test-prep/sat

[2] https://www.amazon.com/gp/product/0525567658/ref=ppx_yo_dt_b...

A good example of this was when Mississippi test takers routinely outscored those in New York taking the test. There were two reasons. The first was that New York required all students to take the SAT while in Mississippi only students interested in attending college took the entrance exams. The other factor was that the ACT was much more commonly used by admissions offices in Mississippi and surrounding states. So out of college bound students, the SAT typically was only taken by a subset planning on attending schools further away, which were much more likely to be Ivy League or other types of schools that attract students much more broadly than the average state school.

I think I took the ACT instead of the SAT, as the ACT was the more popular exam of my day.

The only people that took the SAT in addition to the ACT, were:

1. Applicants to "elite" schools that preferred the SAT.

2. Smart kids trying to get a leg up for large state schools.

3. Pretty smart kids trying to get a bump into a higher scholarship bracket.

If your were anything less than an 80th percentile student, it'd be extremely unusual for you to take both the ACT and SAT. This was also at a Catholic school - most parents pushed their kids to do well academically to make tuition worthwhile.