I think the potential issue is that the driver of a non-emergency vehicle could start gesturing in particular ways that could trigger the autonomous car to think it was an emergency vehicle.

So then you have to teach the car what all the various kinds of emergency vehicles look like, in every jurisdiction. That's hard enough, but then what happens if it's an unmarked police car, for example?