There's the potential for bad actors to plant dangerous training into these algorithms. Hypothetically, someone could turn an entire fleet of self-driving taxis into an assassination mechanism. Every vehicle can wait until a specific individual is spotted crossing the street and fail to stop at a light. The corporation that owns this taxi then will say there was a software failure with the one specific taxi, take it out of commission, and move on with business as usual. A few deaths a year is still less than the death toll of human-operated cars, after all.
Trains also experienced lots of concern, from the legitimate to the ludicrous.
Pointing a gun at a person because you don't like the kind of car they're driving isn't exactly a workable solution.
Politically I doubt that vandalism is a good way to stop self driving cars. Waymo can afford to replace damaged parts, and they could also smear their opponents as criminals, even if it's a minority of those concerned. This is the exact same playbook that those opposed to civil rights used, smearing protests as "riots". Playing directly into that hand is not a wise choice, IMHO.
On the other, if you threaten the employee within the car or attempt to drive them off the road, you should be prosecuted with the full force of the law. There are still people within those self driving cars, and attempted homicide is still a crime.
Edit: being opposed to attempted vehicular homicide is an unpopular opinion. Never saw that coming.