I agree, but also, I kind of feel like real judgement and understanding are a different thing from this "being good at thinking", and that in fact, this is the reason AIs hallucinate in the first place (A generated text can fit the pattern of a good response because its accurate, or because it's bs but has similar words and structure to other texts on the topic. Without real understanding, LLMs can't tell that the truth is important)

It's not directly related to this but I'd like to add that, in my opinion (admittedly uninformed but it makes sense to me), when they manage to, not just reduce the frequency of the LLM stating made up dates, laws etc, or made up facts, but manage to make an AI understand what the truth is and deliberately make responses accurate, we'll have real AGI that will be able to do engineering, accounting, medicine... unsupervised.

And yet I would expect that if it truly was generally good at thinking as you put it that it would be better at chess if you gave it a set of general rules/tactics and have it think through and look forward to decide its moves, but it is quite bad.

How do we decide which version of thinking a model is good at?

A house sparrow (Passer domesticus) doesn't have a worse brain than a human. Nothing does house sparrow thinking better than a house sparrow brain. Not even a human brain.

what about a house cat?

they clearly outhink the sparrow at sparrowing in order to predict what the sparrow will and wont react to, and how theyll try to escape the cat

It's not clear to me that a house cat is better at out-thinking a sparrow than a sparrow is at out-thinking a house cat.

Avoiding predation is a pretty small part of the life of a house sparrow. There is no reason to believe a cat brain would do a good job with handling the demands of flight (avian brains process visual stimuli roughly twice as fast as mammalian brains, for one thing).

But even if we focus on avoiding predation: would a cat brain in a sparrow body (a silly premise, I'll admit!) do better at avoiding cat predation than a sparrow brain in a sparrow body? I doubt it!