Right. The driver would be in control in the scenario I'm describing, just like they are with lane assist, cruise control, or closest to stealth self-driving, auto braking and swerve. I guess it would be better described as the driver would always have the illusion of control.
The automated parts would kick in when necessary and they would be increasingly intrusive in their warnings and modification of driving until the human is only needed for identifying source and destination.
The aviation people hit this problem in the 1990s, as autopilots got more comprehensive. Watch "Children of the magenta", a talk by American Airlines' chief pilot on this.[1]
[1] https://www.youtube.com/watch?v=pN41LvuSz10