At least the machines offer the following hope: even if unbiased humans are rare among paper-grading teachers, those humans can be used to train the machines, so then bias-free or lower-bias grading becomes more ubiquitous.
Basically, the system has the potential for systematically identifying and reducing systematic bias. A computer program can be retrained much more readily than nation-wide army of humans. Humans can be given a lecture on bias, and then they will just return to their ways.
The bias comes from the human-generated training data in the first place; the machine isn't introducing its own. For instance, the machine has no inherent concept of disparaging someone's language because it's from an identifiable inner city dialect. If it picks up that bias, at least it will apply it consistently. When we investigate the machine, the machine will not know that it's being investigated and will not try to conceal its bias from us.
On the other hand, eliminating bias from humans basically means this: producing a new litter of small humans and teaching them better than their predecessors.
AI certainly has a lot of potential for bias, but claims that AI bias is somehow worse than good old human bias always seem shoddily supported (Note I'm not claiming it's untrue. Just that it's never been shown to my satisfaction, which is not surprising given how quickly AI is changing. It may well be true.)
Well, AI bias can combine sampling bias with human bias. Like say we train the AI with the output of only 10 human paper graders, all chosen from the same school district.
Due to the sampling bias, that data could create markedly more (or less!) bias than the entire population of human paper-graders.
The resulting AI will ideally mimic those 10 humans, though; it shouldn't show more bias than that group. If those 10 are flaring racists, and grade accordingly, the AI will be the same. (In fact, we hope that it will be the same, if the algorithm actually works in mimicking human grading.)
That's the problem - there is seemingly no shame these days. People involved "saved time and money", got paid and that's it. "If I didn't do it someone else would" and all of that.
I think you have misunderstood the parent, who is asserting that the machines scoring these essays typically give lower scores to members of these demographics. This does not, by itself, mean the entire system is biased against those groups.
Whether or not the groups are, overall, systematically benefitted or harmed is not relevant to the injustice this article says exists.
An essay could be different from the reference standard because the standard is an example of good writing and the essay is not. Or it can be different because the author has a cultural, regional, gender, or developed background that imparts a different style than anything in the training corpus. Mistaking the two is very, very bad.
And “between 5 to 20 percent” of essays are randomly selected for human review.
So the takeaway is that if you’re one of the 80-95% of (typically black or female) people who the machine scored dramatically lower, but are not selected for human review, your education future is systematically fucked and you have no knowledge of why or how to change it.
Absolutely reprehensible. Anyone involved in the creation or adoption of these systems should be ashamed.