I think people fear the absence of control, and the absence of understanding how flight works; even though there is a lot that can happen to you driving, you or a person you trust is usually the pilot operating a vehicle who's physics you are familiar with.