In practice your assumption doesn't pan out. Humans follow the directions of almost anyone standing in the street waving their hands. For example, a pedestrian in the crosswalk may stop and wave a car on ahead of them. The car is supposed to stop and wait for them to exit the crosswalk.

By waving their hands, the pedestrian can actually be held accountable if this results in an accident.