Legitimacy isn't really the problem, it's the circumstance of a human being in front of your car that prevents you from moving forward, and beyond getting out of your car and talking to them, or just waiting, you don't have much of a choice in what options you have.
I agree with the intent of your point, but what's the alternative? You can't just keep driving if someone is standing in the middle of the road.
In the case of an illegitimate request, a human driver might question the request and take a different action instead.
Are you being directed to run a red light? If you judge that someone is maliciously asking you to do so, you probably will choose to remain still instead.
Are you being directed to stop? If you judge that someone is maliciously asking you to do so, you may choose to turn around, reverse, turn into a parking lot or otherwise remove your vehicle from the situation. You don't have to run people over to refuse a command to stop.
And if you were completely blocked from moving, you'd probably call the police.
My point is that "always follow hand signals" is trivially problematic. I would expect that if self-driving cars followed any hand signal they saw, you'd see kids doing it for the lulz on the sidewalk in front of their school. ..."traffic jam challenge". After all, someone is less likely to act maliciously towards a human than they are a machine.
The more important test scenarios are life threatening or unreasonably disruptive situations rather than harmless pranks.
Nowhere near perfect, but that makes it a lot harder than just getting a spike strip or a hand gun or a demolition derby car. We’re talking about people willing to risk a decade in prison to “do criminal things to a car and/or passengers”. How much work do you think they want to put into disabling a car with a remote? Why not just wave a gun in the driver’s face at a stop light?
Imagine a teenager, dressed in street wear, who jumps into the middle of a street to stop traffic in a way that has no explanation given the scene, while their friends laugh on the sidewalk as they observe.
Or a group of men in masks stopping a car in an alley as their accomplices surround the vehicle.
Any human driver will recognize these scenarios as illegitimate. It isn't just the act of directing traffic that is judged in isolation.