this is bad, but not quite as bad as it sounds. most of the time, you only need to stop as fast as the car in front of you. it's pretty uncommon to encounter a stationary object in the middle of the travel lane. in fact, outside of driving on surface streets in the city, I can't remember the last time I had to avoid a stationary object in my lane.
I'm much more worried by how closely people follow the car in front of them, regardless of visibility. many leave barely enough room to react at all.
While this usually will be fine, there are definitely issues when something stationary does pop up.
While caravan-ing to Yellowstone with 3 vehicles, all traveling in the center lane of the freeway, we encountered a small car with a passed out passenger in the middle lane. My bro-in-law swerved with some room to spare, immediately behind him I swerved with basically zero room to spare, and my father behind me (luckily for them in a Suburban, but bad for the man in the VW) had no chance- I was immediately looking in my back mirror knowing what I was going to see.
There is a decent likelihood that a machine could have swerved in time, but nearly zero that a typical human could/would have in our typical following patterns.
Humans route around inefficient practices. Just as human driving speeds are typically unaffected by posted traffic speeds, they will optimize for their typical experience over written codes for how they drive.
One example is that there are a number of small towns with two way streets that are parked on but only the width of two cars. That creates bottlenecks where cars can only travel in one direction at a time. This might seem like a disaster and it certainly wouldn't work on a busy city street, but in these locations everyone adjusts and when two cars approach a choke point from opposite directions drivers are really good about being cautious and pulling to the side to allow the other party to pass.
There are probably thousands of these local quirks around the world. Handling all of these situations effectively in a fully self-driving car will take an advanced AGI.
A object would need to be substantial enough to cause an accident and then roll into or fall onto a highway. That’s far from a 1 per million miles of driving situation. Remember, something falling off a truck would also take a while to slow down.
But human drivers hit deer all the time, and it's often unavoidable. An autonomous driver is probably more likely to miss a deer than a human, due to substantially better reflex time.
To me, every brown mailbox with a white reflector could be a deer coming to the road, to infrared, with a larger lens, it should be able to tell a lot better.
Autonomous driving doesn’t need to be perfect because we don’t need it. With sufficient alternative systems the only reason for driving will be for hobby (and we don’t want that automated anyway) and heavy load work (like agriculture, mining, or logging) which is already heavily automated.
I'm not walking halfway across the country just to visit my mother.
The sum of these alternatives will almost always outweigh driving in terms of benefits with a notable exception of convenience. So if you are willing to sacrifice convenience when you want to visit your mother, you will almost certainly get there faster and more economically (with the right systems in place) then driving.
With that said, yes there are faster ways (albeit still less convenient) of getting you outside of said city. You might have to change your mode of transportation a couple of time (I said it was less convenient) but it will still be faster with the right systems and infrastructure in place.
1: This is not an unfair scenario because this is already almost the case for all of the non-alternatives.
Kiss an extra two hours of your day goodbye just to get to and from work (assuming you have a job). Going to the doctor is a nightmare (assuming you have medical insurance). It sucks a lot, unless you're living in a city so dense that cars are impractical.
I hope your local politicians agree with me that expenditures going into making these alternatives are money better spent then waiting for the technology to dedicate highway lanes for autonomous vehicles.
That is I hope they agree that your situation of not being able to go to the doctor within a reasonable amount of time takes precedence over people wanting to sleep during their 8 hour highway trip but are unwilling to take the bus for some reason.