but not lidar data, which is why I agree with your statement that google is well-positioned. lidar is expensive now, but I expect self driving cars to be the reason that changes.
Cameras give you rich intensity information: you can easily identify road markings, signs, etc provided the scene is well illuminated. Depth from stereo is OK, but struggles if the scene is featureless. Cameras perform about as well as your eyes in inclement weather. The idea that cameras don't work at night isn't entirely true, because your car should have headlights.
LIDAR is used for robust distance measurement. You can make spot measurements on pretty much any non-specular (shiny) surface. It's active so it works without external illumination (so you get 360 3D vision at night, not just where your lights point), it's accurate enough for driving (cm-level at 10s of metres) and by paralleling sensors you can get realtime performance. The Velodyne system uses 64 rx/tx pairs for ~1mpt/s. In practice you get around 20k LIDAR points per camera image because those 1 million points are spread over a hemisphere and your camera is imaging at 30 or 60 fps.
You can do weird tricks with RADAR though, like the two-car lookahead thing that Tesla has implemented. You can do that with LIDAR if you're looking at a mirror (you'll measure the distance to thing in the mirror), but typically multipath returns don't have a high enough SNR to be useful.
If a car can self-drive anywhere, but will refuse to take control in an empty street covered in snow, I guess that's not a fatal flaw.
It doesn't necessarily need to be absolutely featureless: more specifically stereo matching suffers when there is local texture that is not sufficiently unique along an epipolar line (normally we use rectified images so epipolar = along the image rows). For a concrete example, if I showed you (or a computer) a small patch of road (< 15x15 px square) and told you to find its location in another image, you would struggle because of the ambiguity. This happens all the time 'in the wild'. Cars are shiny, which means specular reflections everywhere; global illumination differences are fine, but local differences cause problems. Matching surfaces like glass is also hard. Someone else mentioned the sides of artic lorries.
LIDAR avoids a lot of potential confusion, but I'm not suggesting that it's a catch-all. It's time consuming to scan and the data are sparse. The best systems (should) fuse data from all the different sources to maximise confidence.
You won't get by on LIDAR alone though. You'll need cameras for stuff like identifying road lines and signage.
I don't understand why the full monty has to be in version 1.0. Give me a good automation on the low hanging fruit, and let convergent evolution of society, infrastructure, and technology iterate to the more difficult use cases.
As automation becomes better humans will be paying less and less attention to the road, making this technology somewhat dangerous in the interim. There has already been a fatal accident involving a Tesla on autopilot where the driver was watching Harry Potter.
* Predictable termination. The user must affirm that they are ready to receive the car. :: User takes over.
* Operator emergency. The user is unresponsive or active indications are that they require emergency assistance. :: auto-park in the nearest assistance area, (video monitored), emergency response crews also en-route.
* Unpredictable catastrophic failure. Internal/External doesn't matter. The vehicle is no longer responsive to the global mesh and a timeout condition occurs. Emergency response is dispatched to the area of incident automatically.
The 'driver taking over' in your scenario would be that final case, however most of those incidents could either be detected and minor failures avoided/tolerated or binned in to the second category. Any other cases are extreme and would result in a cascade anyway, likely even if the human were paying attention as normal.
Thus, the low hanging fruit of travel on the freeway IS the low hanging fruit. Developing a space reserved for automated vehicles and preferably app assisted ride-sharing/carpooling.
Humans can see the scenarios below coming from enough distance to mitigate the problem and AIs have not yet demonstrated equivalence:
1. Slow/erratic person or vehicle suddenly veers into the path of travel. AIs so far tend to just be ultra-safe here and stay far away, which is not the same thing as understanding the situation.
2. Construction zone path of travel is suddenly obstructed. (I'm picking a major scenario to exemplify a class of scenarios where humans surpass AIs still in predicting their environment.)
3. A vehicle ahead performs an emergency maneuver, which communicates unseen/undetected driving hazards. A human can reason from what the vehicle did to what the hazard might be, and immediately begin mitigating the hazard. AIs have not been very forthcoming with detailed behavior descriptions, but they all appear to still be mainly designed as advanced control loops (that is to say, mostly stateless). And yes, it's going to be a legal morass when AIs begin to follow "rules of behavior," since there will always be exceptions.
Building the car itself, creating a ridesharing network, finding parters, swaying consumer perception and even regulatory challenges are going to be walk in park in comparison.
Google/Waymo is well positioned to be the first to solve this problem, since they have been working on this for a decade and because Google has a big advantage in Machine Learning and Computer Vision research. On the other hand Tesla already has cars on the road collecting data.