>You can maybe identify "traffic light", "car", "pedestrian", and "deer" with machine learning. That's just used to guess what they're going to do next. It's not used to decide if they're an obstacle. Obstacle detection and avoidance is all sensors and geometry.

Honestly, getting them to work to this basic level is the "easy" part. Real world driving conditions in urban environments are suboptimal at best. You're not only going to need a system that can navigate around and not hit things. You need it to be able to dynamically route around road-construction, do clever things to find parking.

Roads in most places aren't that well standardized. It takes a lot of small judgement calls for a taxi to actually do its job. Uber and Lyft still have trouble dispatching people to pick you up at your actual door!