News for hackers. Stuff that matters.
Re:New Test for Computers: Grading Essays at College Level by danso on Thursday April 04, 2013 @03:30PM (#5494728)
So, for this to be even a feasible solution, EdX needs to show more proof of concept...like, here's a sample question, here's the 100 sample answers that were used to machine-learn against, and here's how the auto-grader graded these 10 different answers (both good and bad).

Why should anyone have faith that EdX has cracked the perfect mix of machine learning and NLP and other associated technologies needed to provide accurate assessments of essays? Even Google from time to time has trouble guessing intent. Wolfram Alpha even more so. If the engineers at these companies can't get it always right -- and it's not just engineering talent, but data and data analysis -- why should a school entrust one of its most important functions to EdX?

Grading is something critical to get right, not just, "almost there." Think of how much time you spent arguing with a profesor that your score deserved an 8 instead of a 6, enough points to bring you from a B to a B+ for the semester...Think of the incentive to do so (career prospects). If the machine is ever wildly off just even in one case, would you ever take its assessments as gospel? Multiply yourself by 20 or 50 or whatever a usual professor's lecture load is, and now a ton of lag has been introduced into the grading workflow.

Obviously, there are ways to mitigate this. One would be to write questions that are so narrowly focused that there are very clearly right and wrong answers...Which of course raises the problem of: why not just give everyone multiple choice tests, then?

The sad thing is is that even if these machine graders were empirically better than human graders, they can't just be better, they have to be almost perfect. If a plane's autopilot failed, on the whole, 1 out of a 1000 times compared to 5 out of a 1000 times for human pilots...how much more pissed do you think victim families are going to be when they find out their loved ones died because of a algorithmic malfunction rather than just pilot error? People, for obvious reasons, don't like thinking of their work or their destinies being defined by deterministic machines...so if those machines aren't 99.99% right, then the fallout and pushback may cost schools more than the savings in professor-grading time.

Re:New Test for Computers: Grading Essays at College Level by jurassic on Thursday April 04, 2013 @04:22PM (#5495051)
> Think of how much time you spent arguing with a profesor that your score deserved an 8 instead of a 6, enough points to bring you from a B to a B+ for the semester.

I spent zero time doing this, and I never even realized how widespread it was to try to negotiate grades until I helped TA a few classes. Now I see how much professors hate the students who do.

Seriously, what job is going to care that you got a B vs B+ in a class? If that subject is really so important to their mission, they are probably looking for A+ candidates anyway.

Re:New Test for Computers: Grading Essays at College Level by danso on Thursday April 04, 2013 @04:47PM (#5495223)
GPA is, unfortunately, one metric that is used to quickly sift applicants. So one B+ over a B may not matter for one course, but can be quite significant over the course of four years.
Re:New Test for Computers: Grading Essays at College Level by hmsimha on Thursday April 04, 2013 @05:27PM (#5495402)
Additionally, certain scholarships require a minimum GPA be met. A B vs. a B- can be a huge difference to someone faced with losing their tuition check.
Re:New Test for Computers: Grading Essays at College Level by rayiner on Thursday April 04, 2013 @05:34PM (#5495443)
I've never grade-grubbed (you can't really where I went to school), but I've had to explain a B versus a B+ in an interview before.
Re:New Test for Computers: Grading Essays at College Level by Ilmesnkie_Jones on Thursday April 04, 2013 @03:57PM (#5494872)
Even if their perfect its not really good enough outside of say standardized testing because they can't integrate what they learn about the individual students from grading their papers and then use it tutor/advise/grade the student.

A holistic understanding of the individual student and how all students are doing in a course is important. It allows the professor to tailor the course or see where weaknesses in understanding are.

Also I would imagine that there are some exceptional essays which because of their unique nature would slip by the grading system. Subjectivity is a plus and a minus of having human markers.

Re:New Test for Computers: Grading Essays at College Level by rollo_tommasi on Thursday April 04, 2013 @03:36PM (#5494757)
Critical precision in grading is only necessary in competitive high-stakes academic environments where everyone is striving for excellence (or at least a shot at belonging to the social or technical elite). If you consider an environment where the point is simply to train a crowd of semi-skilled workers to a level of basic white-collar functionality (which is what most mid- to lower-tier colleges are approaching, and what MOOCs will probably become), widespread use of this technology becomes much more plausible and acceptable.