fhadley
I'm not much of a fan of a lot of the arguments made in Weapons of Math Destruction, but I do appreciate that in summarizing you draw the distinction between the biases of the engineer or (illogically, but oft-claimed nonetheless) the algorithm itself and the data which is used to train said model, and I think it's quite a valuable concept in regards to this particular paper.
For instance, the data set they're using here is fairly small, and while, they did use 10-fold cross-validation, that's still a bit on the less than ideal side generally speaking neural nets, especially CNN architectures, which are usually pretty deep. Furthermore, the dataset itself seems fairly questionable to me. I'm not sure how much I trust the Chinese criminal justice system to adequately adjudicate culpability in the first place, but even setting aside such admittedly conspiratorial notions, it seems rather odd indeed that nearly half of their positive samples are not in fact convicted criminals but merely suspects. I do not find their attempts at devil's advocate persuasive as it's not readily obvious exactly how they used or obtained any of their testing with the three different data sets.
As for the appropriateness of the broader topic, I'm more or less of the persuasion that all questions deserve to be examined, and that provided the work does not cause direct harm, it's hard for me to support a prohibition on examination of a given topic. That said, I do think that the more controversial the question, the higher quality of research required, and, good lord, does this mess fall well short of the mark. Perhaps if there existed a hypothetical criminal justice system free of systemic biases or, more realistically, a method by which to exactly define those prejudices and account for them in the composition of a data set, this could be a potentially useful question to investigate, but even then it seems to me quite unlikely that there's any particularly significant relationship between one's upper lip curvature and criminal disposition.
edit: here's the ebook link neuralnetworksanddeeplearning.com
Like I said though, it wasn't just good times and passive aggressive complaining about first world problems while I read this. I can't even begin to fathom the miserable, pathetic, and generally small, unexamined existence that would lead one to believe that h(s)e is somehow the victim of deep injustice at the hands of those people with their prejudiced technical abilities and natural curiosity! How dare they not level the playing field just because she never bothered to explore the use of code outside the classroom; clearly everyone missed that she's the subjugated one, with a comfortable liberal arts education and regular internet access.
But still, we should all take a moment to recognize the plight of the comfortable, generally satisfactory lives of those among us struggling silently with the burden of "technical un-privilege."