We don't have to be particularly worried about self-driving cars being confused by malicious physical actors. Such an actor could always do something like cut the brake lines instead.