It's not just you. It's a well-researched phenomenon.
I do think there's the potential to enable full autonomy in well-defined use cases while requiring manual control elsewhere. (Still problematic--how do you enforce?)
But the idea that a driver can take control on short notice from an automated system is a non-starter. Maybe you can assume there's an adult with supervisory control in the vehicle but you certainly can't assume they're prepared to deal with urgent emergencies.
Great point. It's one that not enough people are emphasizing.
I need to be physically engaged in the driving experience or I won't be attentive in the least. It's been a trait since childhood that my mental state is tied to physical activity. A car that keeps me physically uninvolved until it decides that it needs me would be a death trap for me and anyone around me.