I think that's the point of self-driving's limitations. There are many, many ways in which human drivers fail, to the tune of 35k dead per year. But there are also many, many, many ways in which human drivers handle situations safely that would result in self-driving cars either plowing straight ahead or stopping confused, either situation being worse than what happens today. I think these latter situations affect far more than 35k people per year. They might affect more than 35k people per day.
Death vs inconvenience is an obvious trade, self-driving isn't all or nothing. Safety features like auto-stopping and lane-keeping probably reduce that 35k number dramatically if they're more widespread, without taking human hands off the wheel for the other situations.
You are not going to get perfect centralised road status information for driving cars.