> I'd think the author would argue that these racist biases should and could have been caught before going into production

I think people who believe this significantly underestimate the difficulty of solving these problems.

I think that's likely. But the people tasked to solve these problems aren't always given the right priorities. Detecting/classifying the faces of dark-skinned humans is inextricably more complicated than it is with light-skinned humans. It requires significantly more training data, more computational power, and more testing time to reach the same accuracy as a light-skin-only camera and classifier.

We would deride Uber's self-driving vehicles if it turns out their systems can't tell the difference between harmless (plastic bags) and dangerous (jaywalkers, wildlife) objects on the road. Likewise, we can judge a company for releasing a general consumer webcam that fails to fully function for 10-15% of the American population.

>Likewise, we can judge a company for releasing a general consumer webcam that fails to fully function for 10-15% of the American population.

But then you get into an even deeper question 'are minorities/blacks 10% of said consumers of products?'. Poverty, for example, could mean that the product doesn't work for 5% of the buyers, which then begs the question 'how much effort do you put in fixing the problem for a small percentage of the buyers?', especially when the amount of effort is going to be very large/time consuming/expensive. If you are a person looking for racism, you will judge google as racist. If you are looking at it from a position of a business attempting to make a profit on a product, you will not see it as racist.

> But then you get into an even deeper question 'are minorities/blacks 10% of said consumers of products?'.

Maybe run that question by legal first! Protected classes exist for a good reason (IMO). What percentage of a restaurant's business is from people in wheelchairs, yet the law mandates ramp access.

This is very true. I suppose a mirror isn't quite the right analogy for Google, either. It's like film, a snapshot of what it thinks life is like. And for film, there are lots of technical problems for filming black actors, the likes of which had been solved 60 years prior for white actors[0]. It's very possible that the algorithm, captures a version of reality that doesn't show people of color in a good light.

And yes, these problems should be solved, and Google should use its editorial control to do the right thing. I believe it has done so, though perhaps not proactively enough. But these are hard problems, ones that are still not solved in other industries. Web search is still a young industry.

[0]: https://mic.com/articles/184244/keeping-insecure-lit-hbo-cin...

I'd argue that most of the 'algorithms are evil' crowd don't even understand the problem to begin with since many of them do not come from a technical background.

How much does the average driver have to understand about automotive and mechanical engineering before they can form a reasonable criticism about defective/fatal design?

Physical products are much easier to intuit your way around than software. For cars, effectively everyone knows what the engine, fuel tank, tires, and drive selection lever are (though they may call the last one a "PRNDL"). Even if they don't know exactly how an alternator works, they understand that they need to drive their car around after they jump it to recharge the battery.

By comparison, I think most people don't even know the basic architecture of a computer, let alone the applications that run on it. Most people don't have an understanding of the network stack - I consider people ahead of the curve if they know the difference between LAN and the wider internet. Some people even think it's possible to download more RAM...

Driving follows physics mechanics, is a skill and is much more involved than search. Search is designed to be a black box for most people.

But ignoring that, no, I still would not like to see people professionally criticizing processes and products that they do not understand. Some technical knowledge is a must for people who do professional talks or presentations, write books and research papers and teach on the topic.

For algorithms and ML, I would expect at the very least a bachelors in math/CS or the equivalent in work/personal project experience.

criticism follows understanding, so the only honest answer is “quite a lot”

i realize this isn’t widely appreciated or accepted in the age of critical theory, when social desirability leads people to complain about everything in the guise of “fighting oppression,” but it’s a simple real-world fact that no one solves problems from a position of ignorance

I don't understand. Can't these and other sample search terms be added to the regression tests before new versions of the algorithm are deployed? Then you'd likely see the errors in time to correct them.

I think it's more that we think that they're not even bothering to test those things, because they're not thinking about it.

How would you test something like this, if not aggregating user reports? Attempting to preemptively think of every possible thing that could be construed as politically incorrect is a Sisyphean task. It's a lot like the "H8GAYS" airline confirmation code that made headlines [1]. Delta probably blacklisted all profane words, but there are still combinations that can be perceived as offensive. Similarly, I once worked at a summer camp teens that generated users' passwords by picking two words and two numbers (all profanity was blacklisted as were 69, 88, 13, and I think a few other numbers). That didn't stop a camp attendee's parents from getting called in because their daughter was assigned the password "bloodyunhappy12" (the concern was that "bloody unhappy" referred to menstruation).

[1] http://newsfeed.time.com/2013/12/17/delta-airlines-is-very-s...

> I'd think the author would argue that these racist biases should and could have been caught before going into production -- i.e. it doesn't have to be an inevitable cycle of ad hoc public shaming and repentance.

They could have caught more cases before going to production - that's a truism. But when X number of cases get "fixed", there are other N*X cases/other communities/other minorities that won't, and there will be ad hoc public shaming that will caught enough attention to become relevant, and so on. Because the number of things that offend at least somebody is infinite.

My point is that there is no absolute way to tell what's offensive and what isn't - it's all subjective. There is no perfect way to fix them all - not even in theory, because there are conflicting priorities - and public outrage serves as an imperfect way to prioritize the most inter-subjectively blatant. Are there other ways? Sure, but they won't stop the periodic public outrage.