Sure it is, just like a child or someone not very good at reasoning. You can test ChatGPT yourself on some totally novel ad hoc reasoning task you invent for the task, with a single correct conclusion that takes reasoning to arrive at and it will probably get it if it's really easy, even if you take great pains to make it something totally new that you invented. Try it yourself (preferably with ChatGPT 4o) if you don't believe me. Please share your results.
That's a good way to think about it. Treat GPT-4 as having mentality of a 4 year old kid. A kid this age will take any question you ask at face value, because it hasn't learned yet that adults often don't ask questions precisely enough, don't realize the assumptions they make in their requests, don't know what they don't know, and are full of shit. A four year old won't think of evaluating whether or not the question itself makes sense, they'll just do their best to answer it, which may involve plain guessing what the answer could be if one isn't apparent.
Remember that saying "I don't know" isn't an innate skill in humans either - it's an ability we drill into kids for the first decade or two of their lives.
It's not just "I don't know", really. It feels like OpenAI ingrained the essence of North American culture into the model (Sorry North Americans, I really don't mean this in a demeaning way!), as in, the primary task of ChatGPT is supposed to be to make its users happy and feel good about themselves, taking priority over providing accurate answers and facts.
So a young child might answer your question with a response that he heard other people say to a similar question, without actually understanding what he's saying. LLMs are basically this at a grand scale.
LLMs are simply making probabilistic inferences about what "words" (tokenized particles of language, not necessarily individual words from our perspetive) are most likely to appear in relation to other words, based on the training data fed into them.
Their output often resembles reasoning simply because the training data contains large amounts of explicit reasoning in it. But there is no actual reasoning process going on in response to your prompt, just probabilistic inference about what words are most closely correlated with the words in your prompt.
If you're getting what appear to be reasoned responses to "novel" prompts, then one of two things is likely happening: either (a) your prompt isn't as novel or unique as you thought it was, and the model's probabilistic inference was sufficient to generate a valid response without reasoning, or (b) the response isn't as well-reasoned as it appears, and you are failing to notice its errors.
If you want to genuinely test an LLM's ability to engage in reasoning, try throwing a complex math problem at it, or a logic puzzle that trips most people up.
Just make up your own.
Therein lies the mistake. Too many people assume ChatGPT (and similar LLMs) are capable of reasoning. It's not. It is simply just giving you what is likely the 'correct' answer based on some sort of pattern.
It doesn't know what's wrong, so it's not aware it's giving you an inappropriate answer.