Maybe, as you say, it's feasible today or soon to better handle brief distractions but once you allow that it's probably dangerous to assume that people won't stretch out those distractions.
Which means people are either paying enough attention or these self driving systems are quite good. My suspicion is it’s a mix of both, where people tend to zone out in less hazardous driving conditions and start paying attention when things start looking dangerous. Unfortunately, that’s going to cause an equilibrium where people pay less attention as these systems get better.
Do we? Where does that come from? The data Tesla provides is hopelessly non-representative because it makes the assumption that the safety of any given road is independent of whether a driver chooses to switch on the system there.
Comma.ai makes the monitoring more strict when the system is less certain or when in denser traffic.
You cannot, that's the simple truth. You're supposed to focus on the road anyways and should be able to take over once any sort of autopilot or assist system starts working erroneously, yet in practice many people simply assume that those systems being there in the first place mean that you can simply stop focusing on the road altogether.
It feels like the claim of "fully self driving vehicle" is at odds with actual safety, or at least will remain so until the technology actually progresses far enough to be on average safer than human drivers, moral issues aside. Whether that will take 15, 50 or 500 years, i cannot say, however.
That said, currently such functionality could be good enough for the driver to take a sip from a drink, or fiddle around with a message on their phone, or even mess around on the navigation system or the radio - things that would get done regardless because people are irresponsible, but making which a little bit safer is feasible.