That is a recipe for easy sabotage. I think any self-driving car can accept inputs from all sorts of senors and/or peer devices, but at the end of the day it should "trust no one".
Inter-vehicle communication is not immune from that also. But it is a necessary condition, in the same way the air traffic control towers help orchestrate flight and access to airspace.
those comms are controlled by humans, and they are not hackable. If you want an air traffic controller to transmit dangerous instructions to a pilot, you must kill the good air traffic controller and replace him with a saboteur.
Did you not see Die Hard 2?
I don't think you can compare human cognition and driving abilities to current AI/Hardware, since the roads and all the information to drive are geared towards humans.
This being said: I do believe fully automated cars will come, but the 2 conditions I mentioned are prerequisites (in my opinion).
Visual identification algorithms are not fast enough to keep up with driving, let alone distance, direction, and speed inference. Speeding them up is a largely parallel job, but cars don't have the space/power/temperature-controlled/vibration-free environment to support the necessary hardware.
1) Inter-vehicle communications is standardized. This allows for each vehicle to communicate to each other and signal important metrics (speed,direction,intent...), also allows to complement each vehicles 'knowledge' and context of what is going on on the road at any given moment. Communication only needs to happens for vehicles clustered in a given radius (depending on speed and flow of traffic).
2) A critical mass of autonomous vehicles are on the roads.
Once you have these 2 conditions it will be easier for the 'swarm' behavior and flow of traffic to be automated fully. It will also remove a lot of the 'unknowns' stemming from human drivers and smoothly direct traffic.