I think the critics see the amount of bugs in code that has been tested much more than self-driving cars and have very reasonable doubts about giving over something as important as driving. Even without (say) foreign actors trying to hack the cars or jamming the sensors, they seem to keep driving into stationary objects all on their own.