I'd be willing to bet some money on it being simply the result of some employees saying, "AI ethics is super important. We feel strongly there should be people working on AI ethics to make sure it's not racist or sexist". And anyone who objected was yelled at with words to the effect of, "are you saying ethics isn't important? do you not care about ethics?" until opposition was silenced and management felt it was a small sop to quiet the ranks.

The right approach to this would have been to say to those advocating for it the following:

"Ethics is of course tremendously important. That's why we carefully select our workforce and insist on the highest ethical standards from everyone, in their work. Ethics is not something that can be compartmentalised or delegated to a separate team. Ethics underlies all we do. If you see something you believe might be unethical, please raise it with your manager so it can be resolved company wide. Until then, return to work."

Creating a dedicated ethics team was a dumb idea because it can only possibly end like this: you hire people and tell them "your job is to ensure we are ethical" and they're not going to sit back and say, you know boss, I sniffed around and everything seemed above board. Nothing for me to do here right now. I'll check back in a month. Of course they're going to see unethical stuff everywhere they look, regardless of merit. Their salary literally depends on it.

Oh, no doubt. But that scenario would be rooted in this larger context of how we generally relate to this topic.

Creating a dedicated ethics team was a dumb idea because it can only possibly end like this

I agree, which is why I say "it's probably stupidly conceived."

I find social stuff fascinating and I've had the following college classes: Intro to Psychology; Social Psychology; Negotiation and Conflict Management. I also raised two fairly Aspie-ish sons and I've had to explain social things to them in very exacting terms, which forced me to really up my game on being precise in order for it to be useful to them. My sons and I still look for things that meet our rigorous standards for "How do you know that's actually accurate?" because social phenomenon are inherently hard to test: People behave differently when they know "This is a test. This is only a test." than they do in real life (which is why I think UBI experiments are useless: These people know the study will end and the money will stop rolling in, so it doesn't tell you how people would really behave if we really instituted a UBI).

Anyway, I find social stuff fascinating and people routinely interpret my social observations as being ethical or moral judgements and I get this reputation for being a real judgy bitch, basically, even though I"m a lot less judgy than most people. But when push comes to shove, "ethics" is generally about determining which behaviors protect one or both of two things: individuals from group abuse or groups from individual abuse ("tragedy of the commons" type stuff et al).

So when I talk about what works or what doesn't, in some sense I am talking about ethics, but I'm usually not trying to be some kind of moralizing priss, which is how that gets interpreted. I'm usually trying to say something like "Objectively speaking, you can't be both heavy and light at the same time. That fundamentally doesn't parse." and then someone is all up in arms about "Don't talk about my momma!!!! grrrrr!" type stuff.

So I feel strongly that what they really were probably wanting to study is "How do we do AI well in ways that interact with this social stuff like racism?" and out popped the word ethics because that's how people think of that. And it was a terrible idea on the face of it because of the inherent assumption of judgment rather than a more objective framing about studying AI and social phenomenon and how does that work? When you start using AI to do things like parse faces, what are the practical pain points there that we need to be aware of and try to address?

But I think most people can't readily separate that out in that way, which is part of why I am so often in hot water for saying "Well, actually, I think that simply doesn't work because...." And when I say that about social phenomenon, I am often de facto talking about "human ethics" whether I want to be or not and it's very often a giant pile of manure that I've just stepped into.

A subtle AI risk is simply the centralization of the decision making process. This leads to the reinforcement of some subset of biases, until they completely overshadow alternative options. The system becomes rigid, having lost its day-to-day individual level feedback, thus its capacity to adapt to new circumstances. Sooner or later, it will fail.

See also 'too big to fail' and 'central planning'.