Your point needs to be emphasized more. Today is the worse AI will ever be, and it's still better than the average person. Recursive self-improvement will make it even better, maybe on orders of magnitude. When you task an AI, it will be able to walk upward abstractly to ask "is this even a good approach." AI's are good for compartmentalized tasks now, but the architectural tasks will probably be improved in the future.

I can imagine the meltdown when AI starts telling its users "that is not how it would work best, consider this ____" and they rant on about their need for and constant use of their "I don't care, do what I say" clipboard loaded paste.

I'm not generally sold on the "worthy successor" or "human disempowerment would be good" arguments (seems like jumping straight to Hard Mode of alignment when we've not even read the tutorial) but this is one of the ways it might be best for the AI to just… disregard what we say and do something else instead.

We'll get to a point where the best practices are to "tell AI what you want to do, and allow it to tell you how you're going to do it".