>>I believe that law should be repealed, because it would lead to fewer people dying on the roads

Citation needed.

Counter argument - vast majority of males in prison for murder in US are black. Therefore, if we stop anti-discrimination policies in US, that will lead to saving more lives. I mean, statistics don't lie!

Or, maybe, just maybe - the statistics omit a lot of data that would allow us to narrow down what makes men claim more and why there's so many black prisoners in the US?

And that brings us back to the original problem - AI can just come to a conclusion = charge men more for insurance = less accidents! Or....put more black people in prisons = less murders! And the results might be exactly what it expects, so of course it will be declared as a huge success - but it's everything but.

Do you think it should be legal for insurance companies to charge higher premiums to smokers? If so, then you agree it's okay to simply go by the statistics in some contexts, so the existence of some other contexts in which it's not, isn't a counterargument.

I think it really depends on what parameter(s) the AI is optimising for, or weighting the most. If it's efficiency or profit, there are tons of things that make sense financially, but would probably be considered unethical. E.g. we could encourage people to smoke, as it tends to lower the cost of healthcare overall (since they die younger / quicker, on average). That's just an example, the point is that I'm sure there are tons of "truths" from some perspective that an AI might employ, but we'd find unethical.