News for hackers. Stuff that matters.
Re:The Lesson to Unlearn by rglullis on Saturday December 07, 2019 @02:04PM (#21731933)
> If the people aren't motivated, any standardized measure will miss the mark

Shouldn't that be a sign that trying to come up with standardized measures is a fool's errand?

Short of blatant social engineering, I don't see how we can have a system that could improve people's motivation and drive for education. And if we don't have a way to actually change human nature, why do we keep trying to design this system that would magically work for so many different people?

Re:The Lesson to Unlearn by TeMPOraL on Saturday December 07, 2019 @02:22PM (#21732026)
It's a trust issue. It comes from grades being used to evaluate people outside the particular schools they attend.

We could give more autonomy to individual schools or teachers in terms of designing tests, but then we'd be facing two problems. One, the grades given by different schools/teachers would not be directly comparable. An A in one school could be awarded only for exceptional circumstances, and in another just for above-local-average. Two, once you start using those grades to judge people in a competitive situation like the job market, you'll have schools selling grades, teachers inflating grades to help their pupils compete against those from schools that sell grades, etc. and the whole system will collapse to uselessness.

Standardized testing mitigates these problems by forcing everyone to grade by the same criteria, but in order to do that, the grading must be done on things that are extremely easy to measure - which removes both individual judgement of the teachers and any relation between testing and understanding of the material.

The more I think about it, the more I feel it all boils down to scale. The grading system needs to be scaled up with the size of the market people compete on, otherwise it'll be either trivially gamed, or deeply unfair to unlucky people.

Re:The Lesson to Unlearn by GCA10 on Saturday December 07, 2019 @02:38PM (#21732126)
The Big Four accounting firms pay huge attention to college grades and it works well for them. Ditto law firms. And in those fields, the mechanical pursuit of grades -- independent of actual knowledge -- models the actual demands of the job pretty well.

Accounting and law firms are looking for people with the stamina and single-minded goal focus to get document-creation tasks done, one after another, to a high degree of "skill." Getting an A on the test and preparing an SEC-compliant bond prospectus are close cousins of each other.

In fields where real-world mastery doesn't have much to do with the orderly routines of getting a 97 on the midterm, employers are less likely to hunt for people who got 97s on the midterm. (John Steinbeck stumbled around Stanford for six years and never graduated. But he got more accomplished as a writer than a whole decade worth of creative-writing majors with high GPAs.)

Re:The Lesson to Unlearn by rglullis on Saturday December 07, 2019 @02:54PM (#21732214)
> It comes from grades being used to evaluate people outside the particular schools they attend.

Are they really, or is just this elaborate illusion supported by most decision makers to give some rationalization to their own selection processes? When you have Harvard being sued for discriminating against Asians, how can we keep a straight face and tell the kids that they need to study hard because the grades matter?

I think this is part of what PG wants to defend in the essay: why do we actually care about grades at all?

Re:The Lesson to Unlearn by TeMPOraL on Saturday December 07, 2019 @05:02PM (#21732859)
> why do we actually care about grades at all?

Because jobs are at a premium and evaluating prospective hires is hard, so employers look for anything they could use to filter and sort the stream of candidates. Education - both the what and the where is somewhat correlated, so that's what's being used.

Re:The Lesson to Unlearn by rglullis on Saturday December 07, 2019 @05:51PM (#21733128)
The "what" and the "where" do not require the "how many points" to work as a filter, do they?

I think I spent too much on this discussion already, but the point I am trying to make is that standard tests are bad both as a filter for selection and as predictor of performance.

While not necessarily agreeing, I do understand if someone says "I will hire someone good from a top-tier school vs a top student from a not-so-good university". The "what" and "where" are definitely useful data points to filter out candidates.

What I don't understand is any employer that says they will hire based on SAT, GPA or any variation of an IQ test. It seems to be so unreliable measures to the point of actually hindering the process.

Re:The Lesson to Unlearn by knzhou on Saturday December 07, 2019 @02:31PM (#21732083)
We have to try, because the alternative to standardized testing isn't some egalitarian utopia, it's nepotism.
Re:The Lesson to Unlearn by mwfunk on Saturday December 07, 2019 @04:00PM (#21732509)
Yeah. I love HN and many of the thoughts expressed on it, but if there's one thing that always jumps out at me as being an incredibly widely held cognitive bias in the tech community, it's a tendency to view certain things in a very black and white, binary fashion, and to treat certain very complicated, messy problems with lots of social nuances as if they were simple problems, if only everyone could be rational about it. Grades and hiring are obvious examples, but you can also see plenty of it in discussions about open source business models and codes of conduct and copyright and so on.

Throughout my life and career I've seen so many statements of this nature from my peers: "the status quo will only ever be a 90% solution at best, therefore we should throw all of that away and just do everything on a case-by-case basis in isolation from one another, then we'd all live in an ideal meritocracy (or something)".

See also any discussion about flat management. "Explicit management hierarchies obviously aren't perfect, therefore let's get rid of them altogether and assume that everyone is awesome and will magically self-organize and do the right thing."

I guess the common theme is, someone is greatly overestimating their own rationality, and projects it onto other people, and ends up with an almost religious faith in the self-organizing competencies of unmanaged people, or the natural inclination of unregulated schools or industries to always do the right thing even when it's not in their own self-interest.

IME there are lots of fundamentally messy or imperfect things in life, where any choice you make has negative consequences. There's also a personality type that is repelled by this sort of messiness and will always be drawn towards throwing everything away and starting from scratch, often with a childlike belief that things they don't understand aren't important, and why can't everyone see how obvious and elegant their solution is? IME they're usually wrong about those situations, but I guess I'm still glad people less pessimistic than I am are tilting at those windmills.

Re:The Lesson to Unlearn by throwawaymath on Saturday December 07, 2019 @04:24PM (#21732653)
This is an excellent observation.

The problem isn't hyperrationality per se; it's a lack of intellectual humility and self-awareness. The behavior you're talking about arises when someone looks at a problem, immediately wants to fix it and makes snap judgements to figure out how to do so. Instead, they should dispassionately consider 1) whether the problem is fixable in any meaningful sense, and 2) whether they have the requisite experience, skill and insight to solve it in five minutes of thought.

You could further distill this idea with the observation that a lot of people consider things too systematically, rather than holistically. "First principles" thinking is a powerful drug, which appears to crack open the world and solve every problem. But no problem-solving paradigm can deconstruct away unknown unknowns or inexperience.

Re:The Lesson to Unlearn by closeparen on Monday December 09, 2019 @04:37AM (#21741403)
Exercising power over other people is a serious thing. It's extraordinarily presumptuous to believe you have designed a rule that will do more good than harm. I'm much more worried about the cognitive bias that says that because someone is currently the decision-maker, they must be competent at it.
Re:The Lesson to Unlearn by rglullis on Saturday December 07, 2019 @04:47PM (#21732779)
Sorry, it seems that you are jumping into conclusions that were never warranted. Chesterton's fence is a principle that I learned a while ago and I am not defending that we need to throw everything away.

What I am saying is that what you regard as a "90% solution at best" is actually not a solution at all. Standardized tests are terrible predictors of performance on actual jobs. If standardized tests (or IQ tests or anything that is supposed to be a proxy for measuring performance) shows time and again that it only manages to measure people that are good to "study for the test", then why do we keep trying to use them to measure performance? Is it because it "the only thing we have"? This seems like saying "I am lost in Rome but all I have is a map of Paris, let me use that anyway."

And no, I am not proposing some other solution to replace it.

Re:The Lesson to Unlearn by throwawaymath on Saturday December 07, 2019 @04:54PM (#21732811)
What about that comment made you believe it was referring to something you said specifically?
Re:The Lesson to Unlearn by rglullis on Saturday December 07, 2019 @03:43PM (#21732427)
Wait, what?! Why?!

I am not asking for any utopia. What I am asking is "Instead of setting ourselves for failure every time coming up with some scheme for testing performance, why can't we just use actual performance on the activity that we want to evaluate candidates"?

It works for people in sports. It works for trade professions. It works for traders. It works for performers, entertainers, even entrepreneurs.

Why can't we have an educational system that does not put people on tracks to mold them and instead just coach them to follow whatever path they seem fit?

Re:The Lesson to Unlearn by knzhou on Saturday December 07, 2019 @04:00PM (#21732512)
Because proper evaluation takes time and a bad choice carries a huge cost. In the absence of any objective measures, an employer gets around this by just hiring the son of the guy they golf with. Connections.

Sports works, but only because sports are extremely popular, with tens of millions of people enthusiastically participating on a regular basis, and accessible to anybody no matter their age and income. This is not true for, say, working at a law firm. You can’t take the best clerk from a pool of a thousand kids who have been clerking since elementary school. It’s either who you know, or who got good grades and test scores.

Re:The Lesson to Unlearn by rglullis on Saturday December 07, 2019 @05:35PM (#21733039)
> In the absence of any objective measures

One would argue that this is the current status.

How objective are, e.g, IQ tests when I can take two different tests in a span of two hours and get from 128 to 146?

How my grades in Computer Engineering can be used to predict my performance as an Engineer in a country where 85% of the graduates would work in non-technical and more profitable fields?

If a company looking for a Quant gets someone with a Master in Math and a good tracking record knocking on their door, would they stop and ask to see if they can get someone with a PhD instead?

You are right that "proper evaluation takes time". What I am saying is that standardized tests should not count as proper evaluation.

Re:The Lesson to Unlearn by knzhou on Saturday December 07, 2019 @05:46PM (#21733100)
Again, the alternative to standardized tests and credentials is hiring based on connections, ethnicity, pedigree, physical appearance, and superficial confidence. These problems are exactly why meritocracy was invented in the first place. The alternative is not everybody getting the jobs they want, it's you being turned away without any consideration at all because the boss already hired their cousin.

> How objective are, e.g, IQ tests when I can take two different tests in a span of two hours and get from 128 to 146?

You can't take two proper WAIS-IV tests in two hours. In fact, unless you have been part of an unusual psychology study, you've probably never taken a real IQ test in your life. There are a lot of gimmicky free things on the internet that call themselves IQ tests but aren't predictive of anything.

Re:The Lesson to Unlearn by rglullis on Saturday December 07, 2019 @06:14PM (#21733266)
You are moving the goal posts. I was talking only about standardized tests, and now you are also including credentials on the mix.

As for the IQ test - it wasn't online and it wasn't a study, as far as I know - but I can totally concede that it was not a "real" IQ test.

Anyway, the story happened over 20 years ago. Part of the driving exam in Brazil included a "neuro-psychological evaluation" and the doctor flagged me for some extra tests. From what I remember I went to the clinic the next week, did one test, did the "neuro-technical" exam and was told to wait for an IQ test. When I got the form, it was the same as I got before originally (different order of questions) Long story short I asked why I was re-taking the test and I thought I "failed" somehow. They actually said they were not aware of the first one and later on found it.

Now that I think about it, perhaps it was some weird psychology test? :)

Re:The Lesson to Unlearn by Kaiyou on Monday December 09, 2019 @11:24AM (#21743872)
It's pretty straightforward to motivate people. You just have to make that what you teach them has value that they can use immediately. For example, to motivate someone to train in self-defence, just beat them up every week. Now learning self-defence will have value in their life. If they don't get beat up weekly, what's the point of learning self-defence? If it's only once a year, then who cares. Spending time every day to counter something that only happens once a year isn't worth it. It's less trouble to get beat up once a year than it is to spend learning self-defence daily.

There just has to be an immediate value in what you teach. If there isn't, there's no point in teaching it. Students will just forget it anyway after the test.