But the problem is that AI can fail in unexpected ways.

Humans are much worse than AI when it comes to self-driving tasks like image/pattern recognition, sure. But they are inefficient in expected ways where measures can be taken like having larger and brighter signs, stronger rules, etc. But what happens when you don't know for sure when it can fail and when it won't?

It's been a few years since people are seeing the magic of machine learning but something like this was just discovered. Are you sure if someone goes in front of a Tesla with a picture like the parent comment quoted, it wouldn't crash and cause harm to the people inside?

So, it just needs to be (significantly) better than the average human. I think it's even better because each time an unexpected crash happens all cars can learn. We can't even teach people not to drive drunk.

Is it that scary to not know how and when it can crash? You don't think about being blindsided at each intersection...

Good unit testing includes maliciously constructed inputs.