Because codifying any behaviour is explicitly justifying any behaviour, and few engineers want to be responsible for signing off on the feature to run over Grandma.

A workaround thus far has been to abstract the problem into small enough pieces that ARE palatable to sign off on, as your comment shows. "Minimize the number of Grandmas run over" is a different framing than "Should we run over Grandma?".

It won't be random, it will be based on what they feel is best. It's immaterial in this case, because people will blame the AI as a whole when someone dies (even if a death was unavoidable). This is not an option with you the driver, because banning people from driving means nobody can drive a car at all. So the responsibility is shifted towards smaller details, so folks can feel safe in the knowledge that they may drive and they just have to do "nothing wrong". Even if that wrong is ill-defined and some situations have only "wrong" solutions.

Humans also directly suffer consequences for those actions. They show remorse and suffer emotionally attempting to grapple with the outcome of their choice. The legal system takes remorse and suffering into consideration, as it is designed to do.

Do self-driving engineers personally commit to be punished and suffer remorse for their algorithm’s choices? And before you say “it’s not fair, the CEO is at fault!” think about who’s writing the code. The CEO doesn’t make the self-driving car possible, the engineer does.

> If AI drivers generally have less accidents and in the few cases left behave like humans, wouldn't that be a win?

Yes. I think that's a big part of why it's not necessary to "completely, once and for all solve" ethical problems to automate things that might run into them. One could easily argue (and people of course have) that it's also immoral not to take measures that will reduce accidents, which I'm quite sure will happen with AI drivers in the not too distant future.