You're never going to get a one-minute warning about stupid people doing stupid things. Like when some idiot decides to play Frogger across four lanes of traffic because his car stalled out in the median. I suppose we could just accept some collisions in those cases but it would be a mess.
Certainly, the car needs to be able to recognize conditions where it needs to turn over control with, say, one minute warning. And we're not at that point today. But one can imagine things like radio beacons for construction zones for example. It's not easy but it seems much easier than the more general case and it's also possible to get too hung up on truly weird corner cases like traffic going down the freeway in the wrong direction.