I haven't seen any evidence of a self driving car accident in which the car had no potential to warn the driver. All the examples I have seen are when a car misidentifies on object in its path as a non-threat. The problem is that breaking and/or swerving are very drastic actions. There is therefore a deservedly high barrier before a system is willing to take those actions. At the worst you could have two systems running in parallel and one is tuned to be much more aggressive in making those decisions and that information is communicated to the driver.

No self driving car that doesn't have at least level 4 autonomy allows the driver to not pay attention like you are suggesting. The benefit of a cooperative tech is that it takes mental load off the driver. Like I said previously, there are different levels of mental energy that a person can spend on a task. It isn't like the choices are either 100% or 0% of your attention.