But we evolved to have empathy to help understand how humans act in the cases we don't have complete or good information.

This is why "crazy" people make us so uneasy. They don't fit our mental models for how a human should act. Would you be comfortable driving with a road full of unpredictable "crazies"?

I wonder if we'll ever have the same level of trust with AI as humans if it is still being used at a black box level.