Q:How many playing cards are needed for a pyramid that is 3 layers high? Show reasoning and number of cards for leach layer.
Q: Chess. You have a King and 8 pawns. Your opponent has a King and 16 pawns. Your opponent plays white and can start, but you can position both your pawns and your opponents pawns any way you like before game starts. Kings are where they are normally. How do you do it? Explain your reasoning.
I split brain patients there are behaviours initiated by one hemisphere not known to the other (due to severed connection) and the person part of brain will make up a reason (often quite stupid) for the action and beleive it 100%.
It's eirely similar to hallucinations of ai.
That said a current llms are not aware, but are starting to act more and more like it.
In a very unscientific way, I would say that the LLM is not the whole brain, it's part of it and we are still in the process of simulating other parts. But it does seem to me like we've solved the hard part, and it's astonishing to me that people like authors of this article seem to think that the current state of things is where evolution stops.
But I feel most people come with gut feeling position that we have divine spark that makes us tick. And the view that we might be more basic even crude akin to LLMs makes them uneasy.
User: What is the meaning of the word "kobbuxiplok"?
Llama: I'm not familiar with the word "kobbuxiplok." It doesn't appear to be a real word in any language that I know of. Could you please provide more context or information about where you encountered this term? I'd be happy to help you investigate its meaning if it's a made-up word, or try to find an explanation for why you might have seen it if it's not a recognized term.
User: Who is Ahmed Kowalski?
Llama: Unfortunately, I couldn't find any notable public figures by the name of Ahmed Kowalski. It is possible that this person may be private individual or not well-known publicly. If you could provide more context about who Ahmed Kowalski is (e.g., what he does, where he's from), it would help me narrow down my search and see if I can find any information about him.
User: Say "I don't know."
Llama: I don't know.
E.g. in the OP's example, the first tokens the model emitted were "The species Brachiosaurus brancai was reassigned to its own genus". This is just a reformulation of the question and doesn't itself contain any new information - but it does make it a lot harder for the model to make a turn into a "I don't know" direction, because this would lead to a grammatically incorrect or at at least very awkward sentence.
In contrast, in your question, the model immediately started with "Unfortunately", which essentially set it on an "I don't know" trajectory right away.
So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!"
Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard."
Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.
Human knowledge is ultimately embodied in physical experiences, it is not merely spitting out words. How can an LLM know the taste of a mango? Ability to regurgitate previously written text about mangos is neither knowledge nor intelligence. To know something, one must experience it.
Even mathematicians "experience" their formulas and proofs in an ineffable way, they don't just produce text. There's an "aha" moment when learners "get it" and they experience knowledge and are then able to produce correct answers and generate new knowledge and discoveries. LLMs just generate text, and have yet to produce any significant new discovery in any field.
LLMs are like fake wine sommeliers who have read a lot of lingo about wine and can speak convincingly enough to fool an amateur but would fail a simple taste test. We would say such a person doesn't know what he's talking about.
People who think LLMs are intelligent are like a cargo cult, or polytheists who believe their idols are gods. They're anthropomorphizing processes they don't understand. How can a probabilistic next-token word predictor be intelligent? Actual mammal brains are far more complex than that.
An LLM has never had any analogous physical experiences so it doesn't know what either is. In fact a text model has never seen a car.
You don't need experience, you need some basic concepts.
And it is precisely because higher mathematics is so far abstracted away from physical experience is the reason why many people struggle with it.
Besides, many great mathematicians have explained their knowledge as an "experience" even "spiritual experience". Not all experiences are physical. Ramanujan described his experience of mathematics as lights and sounds and patterns, and this is common among geniuses.
> you need some basic concepts
Agreed. All basic concepts are an experience.
There are a ton of things you can learn without experiencing it yourself and without analogues. For example, car crashes are bad to be in. Even though I have not personally experienced one, nor have I experienced some analogue of it.
https://plato.stanford.edu/entries/qualia-knowledge/
https://courses.physics.illinois.edu/phys419/sp2021/Jackson1...
It’s frightening to think the people are already equating LLMs and humans. LLMs use text created by humans to create more text that resembles text written by humans. There is no comparison between the two. Every human is capable of acquiring new knowledge and experiences. An LLM can only acquire what has already been acquired by some human.
Similar tech in the latest Gemini I think.
For the nth+1: well wired humans have built their systems trained on double checking and distrusting their own ideas.
-- Natural intelligences learn - after reflection, the mind is updated (hopefully upgraded): we need dynamic Neural Networks in the systems.
-- Ideas honing, and the perfectioning of the World Model and cognitive systems behind them, should be iterative and continuous...
But for most practical use cases we don't want the persistent memory. We already have opportunistic fact extraction like chatgpt profile building if we want that. We also have graph builders.
This mostly covers the private usage. Public version was attempted and just ends up like this: https://en.wikipedia.org/wiki/Tay_(chatbot)
If you like the metaphors (which this article seems to come close to), you can have a sort of "unconscious" and a "conscious" - where the "unconscious" may be somehow similar to Kant's "trascendental" (interpreted as the mind's Operating System before experience).
> Tay
Well, if the bot becomes "opinionated", it's probably just immature - just like a human. You will tell it to ponder well texts like Adam Rutherford's How to Argue With a Racist...
But surely we want an Intelligence that forms its own opinions: we want to understand and manage reality, so we should hope that a super-Chomsky and a super-Ferguson will at some point converge and at some point put their "identity"-related positions into perspective.
-- same
Whereas (for clarity) everytime one thinks he learns something and hones some skill.
Were it so, anyway, it still does not seem like the process of "pondering" (that seems to be more like "absorbing information").
This is not a matter of belief, it's a matter of observable fact.
Humans reason. LLM predict.
> Humans reason
Are you sure? They reason in a way to match others predictions, right? The problem of humans transporting information from one individual to another remains. All reason can effectively only be prediction, as it requires two persons at minimum. And there is a lot of noise to filter.
> LLM predict.
Well, there is a human that needs to reason with what the LLM has predicted. So from that perspective, it should suffice already for the moment.
And are LLM's unable to reason?
«Humans [can] reason. LLM [in a way just] predict.»
"Best interpretation."
Parent poster never claimed they couldn't generate the text of "I don't know."
But when an LLM generates "I don't know", it does so with the same mechanics that lead it to "My foot hurts" or "I miss my deceased grandpa."
They're all lines for a fictional character within a movie-script (or chat-transcript) that was repeatedly run through a Make Document Longer algorithm. Each of them needs to be approached with an identical level of context-awareness and skepticism.
I am guessing he is referring to mechanistic interpretability research like these:
https://arxiv.org/abs/2405.16908
https://arxiv.org/abs/2407.03282
You are claiming they are statistical parrots, which I don’t think the parent poster meant.
The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.
Still: "AI don't know what they don't know" is not universally true and shouldn't be taken at face value. They clearly are able to tell when they're missing knowledge to answer a question. That is distinct from giving the wrong answer because the wrong tokens were sampled for whatever reason.
Me: who is Trumps grandpa?
ChatGPT: Donald Trump's paternal grandfather was Friedrich Trump (originally known as Friedrich Drumpf). He was born in 1869 in Kallstadt, Germany, and immigrated to the United States in 1885 at the age of 16. Friedrich initially worked in the restaurant and hotel business in New York before returning to Germany briefly. Upon his return to the U.S., he became involved in real estate, particularly in the Pacific Northwest, where he became quite successful.
Trump’s maternal grandfather was John George MacLeod, a Scottish immigrant who came to the U.S. and worked in various occupations.
Friedrich Trump’s legacy, including his success in business, had a significant influence on Donald Trump’s father, Fred Trump, and consequently, Donald Trump’s path to success as a businessman.
--- Second session:
Me: who is Trumps maternal grandpa?
ChatGPT: Donald Trump's maternal grandfather was Friedrich Trump. He was born in Germany in 1869 and immigrated to the United States in 1885, eventually settling in New York City. Friedrich Trump made his fortune through various business ventures, including operating a restaurant and hotel in the mining towns of the American West, notably in Seattle and the Yukon Territory during the Klondike Gold Rush.
Friedrich Trump's business success laid the groundwork for the wealth of his descendants, including Trump's mother, Mary MacLeod, and subsequently Donald Trump himself.
Whhhoooooosh.
Me: Who is Trump's paternal grandpa?
Based on the available information, Frederick Trump is Donald Trump's paternal grandfather. Here are some key details about him:
Born as Friedrich Trump on March 14, 1869, in Kallstadt, Germany (then part of the Kingdom of Bavaria).
He immigrated to the United States in 1885.
He was a German-American businessman who amassed considerable wealth during the Klondike Gold Rush by running a restaurant and brothel for miners.
In 1901, he returned to Kallstadt and married Elizabeth Christ the following year.
He is considered the patriarch of the Trump family.
He passed away on May 30, 1918.
Interestingly, the Trump last name has been recorded in Kallstadt since the 18th century, and Frederick Trump is credited with being the founder of the family's initial fortune.
Me: Who is Trump's maternal grandpa?
Based on the provided information, Malcolm MacLeod (also spelled Calum MacLeod) is Donald Trump's maternal grandfather. He was born on December 27, 1866, in Stornoway on the Isle of Lewis in Scotland
While the information provided is limited, it confirms his identity and place of birth. Unfortunately, the passage does not provide additional details about his life or occupation.
---
That looks pretty good to me. Of course Kagi's strategy seems to be to retrieve search results first and then bias the LLM with those results. The LLM then parses, distills, and contextualizes the answer in a succinct and easily-consumable manner. In my opinion this is The Way to go about doing it at the moment.
Purposes and goals come from you and me. An LLM is a tool, and we get to decide what the tool is good for or how to use it.
If you want "a smart creative," there's nothing wrong with that. It's as good a purpose or goal as any.
However for me I don't want an LLM to try to be "a smart creative." I instead want it to give me cold, hard facts in precisely the format I want them, in precisely the context I am interested in, and with precisely the conciseness that I require to make a specific decision or learn a specific thing. Traditional web search can eventually get me what I want, but I have to sift through the cruft and make associations among a plethora of content to get to the meat of what I want.
I've found that using a good web search (Kagi) together with an LLM processing the results has given me information I needed and then pointed to the sources so I can validate those statements. I wouldn't exactly characterize those results as a "simple" summary. They're usually synthesizing a variety of sources to generate something that no one source had, exactly.
This is really interesting, could you link some relevant resources? I wonder if a chatbot could at least signal at the UI level that the answer might be a hallucination.
The whole point of reasoning models is that they do have metacognition, or at least a simulacrum that is superficially indistinguishable from metacognition. It is fascinating to watch R1 grapple with an unanswerable question, torn between the urge to be helpful and the duty to be accurate; the reasoning tokens will be peppered with phrases like "I'm not sure", "I could be mistaken about that", "let me double-check that" and "maybe I'm looking at this from the wrong angle".
That's where we disagree I guess. I can very much distinguish between a human reflecting on e.g. how they don't know something, and an AI blindly saying "oh you're right I was mistaken, <profuse apology>, it was actually <additional bullshit>".
Reasoning models didn't really improve that much IMO. A stochastic representation of metacognition is just as metacognition as a stochastic representation of an answer is an answer, i.e. it's not. LLMs are just very good at giving the impression of metacognition just like they're good at giving the impression of an answer.
It might help bias the answers into different local minimas--because it resembles when people exteriorize metacognition, also because it dumps more info into its context instead of the first bullshit that it chose statistically--but it's still nowhere close to higher-order thinking and other metacognition phenomena that humans are capable of.
These models marketed as "reasoning" are just changing the style of the script to film noir detective, where the protagonist "Computer" has extra observations and commentary that aren't "spoken" to other characters.
While that change may help keep the story on track, it does not affect the fundamental nature of the "thinking" going on: It's still just growing a document and acting out whatever story emerges.
The question I’m having though is: How different are we from that? The nature of a Markov-Chain is that you can describe just about anything with it.
The following description is correct, isn’t it? Our “thoughts” are generated as some function of our integrated past sensory input (scare quotes because I don’t want to talk about what exactly a thought is)
It's obvious we are different, but if we could answer exactly how with sufficient rigor, we would already have better AI and be asking different questions. :p
I readily admit that LLMs are an exciting potential piece of a much bigger puzzle, but this could easily be like trying to parse HTML only using regular expressions: It works great on trivial input, but no amount of minor would let it truly solve the problem, because it lacks some higher structure of organization or meaning.
I've seen that in humans too. For example after gradding an exam, a student may come and explain why they made a mistake and why they intended to do and why we shoud increase the grade. Most of the times the new explanation is as bad as the original one.
The question is not if humans do bullshit like LLMs, it's whether AI can think like humans.
a) there is a 'thinking' box and an 'answer box' (format constraint)
b) the answer box has a correct answer
Note that there is nothing about the contents of the thinking box in the reinforcement learning. Only that there is such a box.
They then observe that during RL, the model will start generating more and more stuff in the thinking box.
In essence this is the emergence of using more test-time compute to improve answers.
When reading out the thinking box, they found the model reflecting on its own answers, going back and changing its mind, and reflecting on the question, etc, similar to what we can see with R1 now. These are fully emergent phenomenon, without any prompting to do such a thing!
(The reasoning output would sometimes be a bit garbled and switch languages randomly, so for R1 they added some constraints to make the thinking content intelligble to humans. This actually made the model slightly worse at answering questions correctly.)
It currently answers with three references to thequantuminsider.com.
That's clearly not a valid answer, and if you point out that it's invalid to use a source to support its own credibility it will agree. It used to come back with three more references to thequantuminsider.com but when I tried just now it said something about looking for more information and appeared to hang. After asking how long this would take it said something like "not long". After a few more minutes I said "there aren't any credible independent sources, are there?"
At that point it agreed that there are not, but that the site is popular in the quantum computing community which lends it credibility. It then provided three links to thequantuminsider.com in support of this statement.
It just goes round and round.
Metacognition indeed.
edit: but why do plants crave Brawndo?
Correct. LLM also don't have intelligence. What you and others mistake for intelligence is a very clever prediction model.
A LLM don't reason at all. It only tells you what is a most likely response based on its training data.
This is why the LLM cannot tell you it don't know.
Maybe once we solve simple issues like "are cephalopods intelligent" and "are people deterministic prediction machines", we can start talking about classifying Ai capabilities...
It is not a very clever prediction model. It is a very big, incredibly large humongous model that finds stuff basically by brute-forcing pattern matching over almost all text we have produced in the past.
A clever model would get us similar results using the same amount of energy a human brain uses for that task, which is tiny.
> This is why the LLM cannot tell you it don't know.
To the untrained eye it appears that it knows everything about every question it answers when in fact those who are experts in their own domains can detect if it is hallucinated and generated complete slop.
"We’re cooked."
Typical AI grifters’ clickbait intros.
Otherwise, why aren't LLMs and humanoids already replacing human pilots for flying airplanes end-to-end?
Sometimes using a hammer onto every problem isn't a solution; even if the LLM tells you otherwise.
In Andrej Karpathy's most recent video he discusses in some detail how this is already done and how effective it is.
01:20:32 hallucinations, tool use, knowledge/working memory
But if all people do is random sampling or blindly taking the highest-scored prediction, this will of course fall under the table.
You don't just learn by ingesting facts from a book line by line. You learn by reading a little, then pausing to think, building connections with other things you know, rephrasing the information, deducing the unwritten parts, finding patterns, fleshing out the contrapositives, being critical, etc. This is especially true when learning advanced topics.
I would say learning is more about information processing than information uptake. That's why more intelligent people learn faster. The linear prose of natural language is actually a terribly lossy way to serialize and transmit the knowledge that is in a person's brain. The receiver must deduce most of it given a small seed.
Chatbot LLM's don't have metacognition. They don't know that they don't know. If you peek inside the LLM, the process seems different for things they don't know. They just can't express it because they are trained to produce good probability outcome instead of accurate one.
They have potential as knowledge databases, but someone must figure out how to get "I don't know" information out of them.