I have very similar observation, and while it is amazing at times at helping me with complex tasks, it just isn't like any human. If we rank the tasks by difficulty for human and AI, it just has very small correlation. Also GPT 4 is idiot in conversation, compared to its problem solving skill. I would have expected completely opposite trajectory for AI. eg it's hard for it to get it to ask good clarifying questions. There are lot of cheap tricks in prompting like asking it to act as expert, or prompt for chain of though that works.