I agree that plenty of people have been too cavalier about slapping together some models, predicting something, and calling it a day. On the other hand, it's not like fair sentencing or fair loan recommendation is a solved problem for humans either. There is evidence that, when carefully designed, algorithms can produce more equitable outcomes than humans, for example when deciding who and how to release on bail [1].

So I think we don't want to hand over sentencing decisions to a neural network that nobody understands, but I also think careful use of (simpler) machine learning can still improve a lot of our decision-making. The question is how much these decisions improve over what we have now. There's not exactly a surfeit of wise, highly trained ethicists who are happy to make consequential decisions all day. Many human decision-makers are quite flawed too, they just get to hide behind the opacity of being a human rather than an algorithm.

To that end, a whole "fair machine learning" field has sprung up over the past few years to study this. There are like a dozen papers in the area at NeurIPS and ICML every year. There's some progress.

[1] https://www.nber.org/papers/w23180

Besides, the options are not restricted to "a machine decides everything" and "a human decides everything".

We can use machines to guide the humans, finding flaws or biases, recommending further analysis, and in a lot of other ways while still keeping the decisions made by humans.

This is a bit like telling the whole CS community in the 90s that we need a year off the internet to study ethics because people can use it for child porn, money laundering, and cybercrime.

Should we have?

Parent's condescension drips a little heavy, but anyone building power tools for a species like ours ought to reflect a while on the potential consequences, imho

Yes. The lack of ethics requirements in CS curriculums is a terrible oversight.

Would an ethics course and professional society eliminate child porn online? Of course not. Would it have shaped and possibly tempered social media giants into something less toxic? Maybe.

Do you literally think these people are creating harmful technology because they have had a deficit of ethics courses?

I'm not convinced ethics course requirements do much for a moral living. (And a peculiarly large fraction of those curricula often seems to be concerned with agonizing over whether to push various small or large groups of people on to railway tracks to be run over, rather than trying to stop the frigging train)

The examples you link don't seem very difficult to assess if they're approached with high priority placed on not causing trouble for the innocent. Also, presumably the people purchasing and deploying systems like that have usually taken plenty of formal education about ethics (and possibly philosophy), and yet there seems to be plenty of demand. Why is that you reckon?