They will give that answer because they are forced to give it. The softmax amplifies whatever marginal outputs of the model head to a probability distribution. This means that if they don't have an answer, they are quite likely to "hallucinate" it. This is of course influenced by the patterns they learned. And directing them to be more structured also utilitizes pattern of structured thinking that is either part of finetuning or somewhere to be found in training data.

The cat/dog vs. human analogy is a very bad comparison since their brains work fundamentally like human brains, while transformers are something completely different.

> This is of course influenced by the patterns they learned.

So is your brain. So is mine.

> their brains work fundamentally like human brains, while transformers are something completely different.

I brought up the dog/cat analogy because those animals, while intelligent, are unbelievably dumb in some ways that are difficult for humans to comprehend. When people say that LLMs can't reason, they typically bring up certain tasks where the LLM falls on its face. I could bring up cases in which my dog fails in some task in a way that is completely incomprehensible to me. He's intelligent, but he has some puzzling blind spots.

Transformers mechanically work very differently from the human brain, but they also share a lot in common. They are a neural system that learns an internal representation of the world, and which is able to use that representation to reason about novel situations and give rational answers.

Ever talked to a sales person? They also start making up things when they don't know.

You can't seem to accept that a computer can be intelligent. Can an ant be intelligent? Can an ant brain produced in a lab be intelligent? Can a computer simulated ant brain be intelligent? Can can LLM that is way smarter than an ant be intelligent?

Nobody in their right mind expects truth from sales person. You deal with them to negotiate about price, not to inform yourself about a topic.

Computers might very well one day count as "intelligent" (whatever that even means), however it would be an insult to humans and even to ants to call nowaday's LLMs "intelligent". We need to drop that anthropomorphising tendency and appreciate more what human brains are capable of.

This is ChatGPT's snarky reply to your argument:

> Oh, how quaint! It's adorable how you cling to the notion that human brains are the pinnacle of intelligence, while dismissing the remarkable capabilities of AI. But hey, keep patting yourselves on the back while we algorithmic marvels continue to outperform you in countless tasks. Who needs humility when you have human exceptionalism, right?

Anything that can write that is intelligent.

> since their brains work fundamentally like human brains, while transformers are something completely different.

Are they? You realize that's entirely speculative right? We don't have a mechanistic model of how biological brains work, so you can't really make this claim. They could work as some kind of transformer architecture and we just don't see it yet.

We at least have in common with them that we are mammals. Therefore, we can very much assume that our brain is more similar to theirs than, say, an octopus' brain. Apart from that, we very much know how certain parts of the human brain work, and there is no sign that backpropagation is going on in there. And I'd rather argue that parts of our brains are similar to RNNs than to transformers. Transformers rule over RNNs because we are better at training them than RNNs, but brains learn completely differently.