"This is the most important sentence in the article. It points to a problem in the test itself - California's students didn't just get magically stupider."

Yes. Having gone through public education in California the tests were treated as a total joke. They had no weight to your grades and there was zero incentive to do well. Often you wouldn't even get the results. In elementary school it was common for a teacher to say "When you have finished with this test, you can go to lunch" -- well kids would just start filling in bubbles. These tests were also accompanied by a sheet to see if you have tried drugs or sex all anonymous. My entire 3rd grade class did heroin.

The lack of incentive to do well is precisely the point of these tests, and they are designed with this context in mind. The testing regimen relies on the same level of apathy in both students and teachers being present year after year, which provides a nice baseline for statistical analysis/comparison. Incentives mess everything up. See also: NAEP [1]

Source: worked in ed covering this subject, studied assessment in ed school with one of the creators of the standards.

[1] https://nces.ed.gov/nationsreportcard/

In my experience certain children (and adults) -- ones who live in successful, wealthy neighborhoods are completely terrified of the idea of failing, at anything. They will start weeping if they feel like they performed poorly on an exam. Any test, no matter how meaningless, determines their self-worth. I'd imagine these students do disproportionately fantastic on this test. At this point it's culture/personality test, not a reasonable countrywide baseline.

And that's perhaps why they, or at least their family became successful and/or wealthy in the first place?

If you don't even want to be successful and/or wealthy, and you don't try, your chance of becoming so is definitely much lower.

I think the pathology of worship of competition and the credentialing system -- insisting you must be the best on a test that you are explicitly told means nothing is a mental disorder developed due to being raised in certain cultural contexts, not a cause of success -- do you really think breaking down in tears at the slightest inconsequential setback is a good recipe for success in the primarily luck-driven market economy we live in? How many times do eventually successful entrepreneurs fail and keep going undiscouraged?

I don't get why incentives mess test up? I would try harder on graded test, including preparation.

When a measure becomes a target, it ceases to be a good measure.

The more you prepare specifically for a test, the harder it is to determine how much of your score is based on your general understanding of what is being tested, and how much is based on you exploiting the test.

But it seems humans only tend to acquire and demonstrate knowledge for some sort of self gain. These tests would then just tell you the general level of apathy of the students and do not tell you how much they know.

If I wanted to know how well the average person on the street can clean a toilet and my test involved asking a bunch of people for a demonstration without pay nor feedback of any kind I don't suspect I would find a lot of clean toilets. I would instead have learned that people don't like cleaning toilets and gained no insight into their ability to do so.

And the more you discriminate between good students who prepared the test and the bad students who didn't bother, exaggerating the contrast.

If I don't study at all ever, only then is my understanding pure and not contaminated by studying for a test?

If I dont study for the test, then you basically measure how long ago it was when I was revising the material last time (if it was last week I get much better result then if it was month and half ago) and you have even more random in terms of whether my learning coincided with I focused on what would count as "exploitation" of this test or different test. For example, was you focusing on speed during easy exercises or ability to solve hard problems last three weeks? It is pretty much random choice and preparation lets you get back to other mode relatively fast - while without it you get worst results with kids who accidentally did wrong thing and it has nothing to do with understanding of material, just a random chance.

Motivation to actually prepare yourself removes that random.

The exploitation of specific test, especially if you dont understand base material, gets you only so far compared to that.

>They had no weight to your grades and there was zero incentive to do well.

This doesn't seem like that bad of an idea. One of the bigger problems with tests is that they tend to become a terrible metric once people start preparing for them specifically (and test-prep can suck the oxegyn out of real education). If the students and teachers don't care about the test, it is much easier (but still difficult) to make a test that gives useful feedback to the school system. Of course, you want students to take it seriously enough while there taking it.

> This doesn't seem like that bad of an idea. One of the bigger problems with tests is that they tend to become a terrible metric once people start preparing for them specifically (and test-prep can suck the oxegyn out of real education). If the students and teachers don't care about the test, it is much easier (but still difficult) to make a test that gives useful feedback to the school system. Of course, you want students to take it seriously enough while there taking it.

I remember when I was a senior in college, my professor asked me to take a test one day. She said it had nothing to do with my grades and it was to compare the outcome of our college's program with other colleges (BS Math). I still gave it my best shot even though I don't remember if I got to see the result.

I don't like how you need so many licenses to be a teacher. Most college lecturers do not have a background in education and definitely don't have a license and we do just fine. The licenses just feel like a way to limit supply.

>I don't like how you need so many licenses to be a teacher. Most college lecturers do not have a background in education and definitely don't have a license and we do just fine. The licenses just feel like a way to limit supply.

Teaching adults who pay for the privilege of being in your class is VERY VERY different from teaching children who are forced to be in your class.

The latter requires a very special skill set that can only be described as 'managing psychopaths'.

The former just requires you to show up and tell them they're on "responsible for the material" (i.e. they're on their own).

> The latter requires a very special skill set that can only be described as 'managing psychopaths'.

And without exception, every teacher I've ever spoken with (including several in my family) has told me that this skill is something you have to learn on the job. All of them, regardless of what route they took to become a teacher, were unprepared for managing classroom discipline.

And it took all of them a year or two to learn how to manage classrooms (some teachers never learn this skill).

I strongly agree that the licensing is just a way to limit supply. I'm not against unions per se, but in this matter, the teacher's union is a serious impediment to educational achievement in this country. It should be much easier to hire and fire teachers in this country. And teachers should be much better paid.

> The latter requires a very special skill set

I'm going to argue against that idea.

1. The bar to become a teacher is quite low.

2. Most of the teachers I've met certainly didn't not have the necessary skills, but they, and the students endured.

Being an outstanding teacher required a very special skill set.

That 'managing psychopaths' portion of the teacher skill set sure seems similar to that required of babysitters.

Babysitters generally don't need to manage more 15 children all at once (none of which are related or have known each other for a very long time), in a setting that is not their home, for longer periods of time than they are with their families, for the purpose of education and not play.

There are transferrable skills, but they're transferrable in the way that making buildings out of LEGO is transferrable to building actual buildings. There are large differences and a whole host of other challenges.

[I'm not a teacher, but I have a few friends that are teachers or training to be teachers.]

> Babysitters generally don't need to manage more 15 children all at once

Sorry for going off-topic but I come from a small school and I don't understand how it is acceptable for class size to be huge (40+ students in a class is considered acceptable). I mean I get it if the class is led by someone extraordinary like Richard Feynman and everyone wants to be in his lectures but as others have said most teachers are not celebrities.

Plus, I don't think just because someone went through the motions of getting a license thirty years ago and all the certification conferences means that they are decent teachers today.

It boils down to lack of funding and lack of teachers. Lack of new teachers -- as well as absolutely insanely strong unions -- are also some of the reasons why teachers effectively have tenure at schools. In Australia, teachers can be moved around schools (in a given state) as well, which is even more flexible than just tenure at a university.

> It boils down to lack of funding and lack of teachers. Lack of new teachers -- as well as absolutely insanely strong unions -- are also some of the reasons why teachers effectively have tenure at schools. In Australia, teachers can be moved around schools (in a given state) as well, which is even more flexible than just tenure at a university.

So basically we have no friends. The unions are not our friends obviously but the people who want to break teacher unions are also not our friends because they want to profit at our expense.

There's nothing insidious about unions protecting workers that are members -- that's explicitly their whole purpose. More fields should adopt strong unions.

The problem is that education reform is an common empty promise in politics. Education is underfunded in most countries, and better funding (and then more education for teachers) would be very helpful.

This dismissive attitude really pisses me off.

Babysitters don't have to get the kids to DO anything.

Saying these are the same is like saying a manager in a business is only supposed to check if employees showed up and clocked in.

Sure, some managers are that bad at their job, but it is also very clear that their job is supposed to be much more than that.

Same with teachers. Getting 30 children who did not choose to be there to actually DO something is a very different skillset than babysitting.

/rant

That's the ideal. Except that in the status quo, only the good teachers need to get the kids that don't want to DO anything to DO anything, and the practical consequences for failing to motivate the slackers are nil.

Otherwise, there are two basic possibilities depending on the kid's parents. Either the kid is expected to try and DO something (and does), or the parent only expects child care. In the latter case, avoiding willful destruction and interruption is all that is required … tada, babysitting!

The manager analogy doesn't make sense because managers are much more directly accountable for whether the employees achieve anything for the company while clocking in. If managers' mostly subjective assessments were the only tangible output of employee time and other impacts were felt years later and weren't directly ascribable to an individual manager, then maybe the analogy would work.

Are you certain that most college lecturers do just fine? I have heard multiple complaints from students at California public colleges about lecturers who could barely speak English.

That complaint has been around as long as there have been lazy students. When lazy students find themselves in a class taught by a foreigner with a thick accent, they almost invariably blame the accent (barely speaks English) for their own failures, which, as often as not, probably have more to do with students' own laziness.

> they tend to become a terrible metric

Not exactly. When a test becomes study-able, it becomes a good metric for how hard people try and how much time/effort people put into their studies/career. And that is arguably what people making tests were trying to or would want to measure.

Not for a standardized test gauging the understanding of student's knowledge of Math or English.

When was this? I graduated high school in 2002 and felt like the tests were taken fairly seriously.

I went to school in Marin, though, so I don't know how that compares to anywhere else.

I agree, it always seemed to me that most of us didn't want people to think we're stupid. It seems sad that you have to worry that a kid will just fill in whatever so they can go to lunch. That seems like a self-esteem or discipline problem, not a fault of test taking.

For all the complaining, I'm curious what a better solution would be for finding out how much kids are learning.

I’m only a couple years younger than you, but I definitely experienced a significant “trying is for white people” attitude when I was in school in LA.

>It points to a problem in the test itself

Please forgive what might be construed as quibbling, but I'm increasingly dismayed by the doxa that poor scores on tests reflect poor testing practices.

To be clear, testing practices in the U.S. are crappy, but so is the secondary education system in general (at least, compared to much of Western Europe).

More to the point: these results point to a problem either with the test or with the educational system, or with the cultural values surrounding education in the U.S.

My bet, having lived on both continents, is on the latter. I invite the reader to read Assimov's 1980 essay A Cult of Ignorance, as it provides a good description of what I believe to be the fundamental problem.

You're misunderstanding the entire point of my post. I'm not talking about the scores being bad, I'm talking about the scores miraculously shifting downward for every state that uses that test in the last year.

Your comment would have been more credible if you had spelled 'Asimov' correctly.

That's false. A simple spelling mistake, which everyone makes from time to time, does not invalidate his or her comment.

And your would be more credible if it had substance.

At least they haven't gone down the UK route where kids a rigidly prepared for the multiple tests and GCSE /A levels.

There was a recent scandal where half way through A levels (final two years of high Scholl) if you where not predicted to get a B you got kicked out to protect the schools ranking.

Not a testee in these tests but I put C for everything in my day.

No incentive.

Easy solution: make the tests count for a significant chunk of the students' grades. Or at certain years, require a passing grade on the standardized tests to be promoted to the next grade.

First you have to make the case that it's actually a problem. Test scores didn't go up. So what? Is it actually a problem if the scores just stay the same? I mean, they can only get so high before you have to start screwing with the test to get different results. Why not just accept that the same scores year-over-year mean we're pretty much doing fine?

If we're going to have an enforced one-size fits all national curriculum, we can do better than the one we reverse engineer from some stupid standardized test.

Can we?

If we can't, then we have no hope of succeeding with a one-size-fits-all national curriculum anyways, so either way the point is moot.

Yes, boil the entire year down to one or two tests which have been shown to be a poor indicator of knowledge and performance. We'll just start holding back 25% of students. What could go wrong?

As is typical, easy solutions are often bad solutions. Teaching to the test would be the first-order effect, and that is a terrible use of class time.

Holding kids back a grade isn't really a thing they do anymore.

But, why?

I think this is more important:

The numbers are half a percentage-point different from 2016 — down in English and up in math.

Scores didn't really do down, they stayed the same. Why would one expect them to go up? Kids didn't get magically smarter, schools didn't change. It's more suspicious that the schools got better every year under the previous test.

Optimizing test taking took some years, maybe?

Sure, that's possible, and may suggest the current test is better, not flawed.

So what's your recommended alternative metric? I don't doubt there is a good case to improve certain standardized tests, but that doesn't mean there isn't a valuable correlation even in a flawed one, nor that it's better to keep moving the goal posts into more subjective territory to explain away bad outcomes.

They're mentioned in the article. Graduation rates, college acceptance, stuff like that. Some of the countries with the best education systems in the world don't have these kinds of standardized tests.

Also, nobody's suggesting moving the goal posts to explain away bad outcomes. If every school suddenly loses three points on average from one year to another on what is supposed to be the same test, the only common variable there is the test itself. The test HAS to be different from year to year in order not to repeat questions, but it HAS to remain a comparable difficulty.

Graduation rate strikes me as a dangerous metric because it could easily be maximized by lowering the bar. We want to graduate as many students as possible but we should not compromise quality to do so. It seems to me we need metrics on both.

I do admit to not knowing what I am talking about so maybe there is no danger in using the graduation rate metric.

Well the best metric is going to be something like annual pre-tax income a decade after graduation but that will never happen so we'll just need to figure out some kind of composite metric that avoids perverse incentives.

edit: This is not an entirely serious suggestion. See later comment below.

Should income really be the final metric? What about less tangible (measurable?) factors like happiness? If we program all of our students to maximize income do we really make the world a better place?

It's not a metric for the students though, it's for the school. If you average it among all students, you get a reasonable metric for the how the school is doing. Of course, this would likely fail if you applied it to arts schools...

Anyway, it wasn't a completely serious suggestion. It's got some problems. Happiness is a good thing to measure, but you'd want to still do some long term tracking of other metrics. Incarceration rates, income, college graduation rates, higher degrees and certifications, etc.

edit: I make the comment about arts schools having received a degree in film and taken a haphazard path into startups and programming that ultimately worked out pretty well. I absolutely encourage people to go to school for the arts if that's what you want to be doing.

> What about less tangible (measurable?) factors like happiness?

A primary goal of school is making kids smarter and ultimately a functioning adult. Including being employable.

I don't think "happiness" is the job of an educator. The parents, absolutely. The local government? Maybe.

But if "good citizen" is a desired goal, we might be able to measure that. With surveys of alumni about current events, employment rates, financial principles, etc. To measure whether well-equipped, life-long learners are produced.

But I'd rather schools stay out of holistic goals better reserved for families.

Ignoring all that the timeline for that is so long it's not really useful for evaluating any changes because it'll be 10+ years before you have any new data, depending on where the change happens. On top of that it's so broad that it can only really measure the whole system from elementary up through college/trade-schools so even once you get data you can only really evaluate the system as a whole.

Graduation rates and college acceptance were tinkered with in New York to dismal results. Educational administrators were perversely incentivized, and it has showed up in atrociously high failure of local students to test out of remedial courses in public colleges.

Just because the test mean drops does not mean the test is flawed. The educational inputs could simply have worsened, which is exactly what a reliable indicator should show.

Do you have a metric that doesn't require waiting until students are leaving high school to return results? Some of these scores are for primary school students. It's not very useful to their teachers to be told 5-10 years after they teach a student that the improvements in instruction they attempted had or failed to have an effect.

Can you provide a reference to those countries with the best education systems?

Finland has just one, an exit exam at the end of high school, like a mandatory SAT.

The existing test is privately developed, opaque, and the responsible policy makers in the states using it, including California, can't get information on the validation from the maker. And, regardless of previous trends, which varied, every state using it saw a drop in 2017. This is all mentioned in the article.

Either there was a common national effect that dropped real outcomes, all states using the test independently made simultaneously missteps that dropped real outcomes, or the 2017 test was changed in some way which produced the downward movement.

If it weren't for the opacity, it would be easy to eliminate the last possibility.

> So what's your recommended alternative metric?

AFAIK, good test scores correlate well with college success, but don't say much about quality of life beyond that. There might not be a good metric. For a lot of people, one or two great teachers have more of an impact than anything that can be tested for.

It's almost as if the standard of what is standard changes drastically as the world does.

Or the process by which they design and score the test isn't very good.

"Standardized testing isn't a particularly good way to gauge schools' ability"

Maybe not the best, but it's still a measure, however crude, and that it's going down is a legitimate cause for concern.

It's not magic - basic reading and math comprehension, are probably fairly strongly correlated with all sorts of life outcomes, though as you point out maybe not as direct as we would like to think - it still matters.

As messed up and politicized as the teaching industry may be - at the core level, again, it's not rocket science: decent teachers, students who are somewhat motivated and want to learn, a classroom with a blackboard and some books etc. etc..

I get the sentiment, but it's possible that schools have in fact changed a bit since last year.

It's not a stretch to imagine the introduction of new math textbooks causing a change of a couple points, if the books are in fact better/worse.

-> magically get stupider

Anyway, it's actually more than magic at work; kids in grades 3-8 and 11 in 2016 are not the same set of people as kids in grades 3-8 and 11 in 2017.

One thing I'm getting at here is that even if we discount things like demographic shifts, there is the factor that the kids are one year older and are getting the harder tests. The new grade 3 kids being tested are last year's grade 2; the 4's are last years 3rd graders and so on. If you don't teach the kids anything all year, push them onto the next grade and then test them at the next level, they will do worse. Based on this idea, a year-over-year performance should be able to detect a real, instantaneous problem.

Are you suggesting that the entire state education system can suddenly get worse for everyone everywhere all at once over the course of a year?

That seems... unlikely.