E.g. in the OP's example, the first tokens the model emitted were "The species Brachiosaurus brancai was reassigned to its own genus". This is just a reformulation of the question and doesn't itself contain any new information - but it does make it a lot harder for the model to make a turn into a "I don't know" direction, because this would lead to a grammatically incorrect or at at least very awkward sentence.

In contrast, in your question, the model immediately started with "Unfortunately", which essentially set it on an "I don't know" trajectory right away.

So when they see LLMs say "I don't know", their reaction is "it doesn't know what it is talking about", not "wow LLMs actually can tell what it knows!"

Conversely, when they observe a human confidently says something wrong, their reaction is "what a cocky guy" but not "oh so humans are not better than LLMs in this regard."

Because they already believe humans are different and superior, whatever they observe doesn't affect their opinion.

Human knowledge is ultimately embodied in physical experiences, it is not merely spitting out words. How can an LLM know the taste of a mango? Ability to regurgitate previously written text about mangos is neither knowledge nor intelligence. To know something, one must experience it.

Even mathematicians "experience" their formulas and proofs in an ineffable way, they don't just produce text. There's an "aha" moment when learners "get it" and they experience knowledge and are then able to produce correct answers and generate new knowledge and discoveries. LLMs just generate text, and have yet to produce any significant new discovery in any field.

LLMs are like fake wine sommeliers who have read a lot of lingo about wine and can speak convincingly enough to fool an amateur but would fail a simple taste test. We would say such a person doesn't know what he's talking about.

People who think LLMs are intelligent are like a cargo cult, or polytheists who believe their idols are gods. They're anthropomorphizing processes they don't understand. How can a probabilistic next-token word predictor be intelligent? Actual mammal brains are far more complex than that.

An LLM has never had any analogous physical experiences so it doesn't know what either is. In fact a text model has never seen a car.

You don't need experience, you need some basic concepts.

And it is precisely because higher mathematics is so far abstracted away from physical experience is the reason why many people struggle with it.

Besides, many great mathematicians have explained their knowledge as an "experience" even "spiritual experience". Not all experiences are physical. Ramanujan described his experience of mathematics as lights and sounds and patterns, and this is common among geniuses.

> you need some basic concepts

Agreed. All basic concepts are an experience.

There are a ton of things you can learn without experiencing it yourself and without analogues. For example, car crashes are bad to be in. Even though I have not personally experienced one, nor have I experienced some analogue of it.

https://plato.stanford.edu/entries/qualia-knowledge/

https://courses.physics.illinois.edu/phys419/sp2021/Jackson1...

It’s frightening to think the people are already equating LLMs and humans. LLMs use text created by humans to create more text that resembles text written by humans. There is no comparison between the two. Every human is capable of acquiring new knowledge and experiences. An LLM can only acquire what has already been acquired by some human.

Similar tech in the latest Gemini I think.

For the nth+1: well wired humans have built their systems trained on double checking and distrusting their own ideas.

-- Natural intelligences learn - after reflection, the mind is updated (hopefully upgraded): we need dynamic Neural Networks in the systems.

-- Ideas honing, and the perfectioning of the World Model and cognitive systems behind them, should be iterative and continuous...

But for most practical use cases we don't want the persistent memory. We already have opportunistic fact extraction like chatgpt profile building if we want that. We also have graph builders.

This mostly covers the private usage. Public version was attempted and just ends up like this: https://en.wikipedia.org/wiki/Tay_(chatbot)

If you like the metaphors (which this article seems to come close to), you can have a sort of "unconscious" and a "conscious" - where the "unconscious" may be somehow similar to Kant's "trascendental" (interpreted as the mind's Operating System before experience).

> Tay

Well, if the bot becomes "opinionated", it's probably just immature - just like a human. You will tell it to ponder well texts like Adam Rutherford's How to Argue With a Racist...

But surely we want an Intelligence that forms its own opinions: we want to understand and manage reality, so we should hope that a super-Chomsky and a super-Ferguson will at some point converge and at some point put their "identity"-related positions into perspective.

-- same

Whereas (for clarity) everytime one thinks he learns something and hones some skill.

Were it so, anyway, it still does not seem like the process of "pondering" (that seems to be more like "absorbing information").

This is not a matter of belief, it's a matter of observable fact.

Humans reason. LLM predict.

> Humans reason

Are you sure? They reason in a way to match others predictions, right? The problem of humans transporting information from one individual to another remains. All reason can effectively only be prediction, as it requires two persons at minimum. And there is a lot of noise to filter.

> LLM predict.

Well, there is a human that needs to reason with what the LLM has predicted. So from that perspective, it should suffice already for the moment.

And are LLM's unable to reason?

«Humans [can] reason. LLM [in a way just] predict.»

"Best interpretation."

Parent poster never claimed they couldn't generate the text of "I don't know."

But when an LLM generates "I don't know", it does so with the same mechanics that lead it to "My foot hurts" or "I miss my deceased grandpa."

They're all lines for a fictional character within a movie-script (or chat-transcript) that was repeatedly run through a Make Document Longer algorithm. Each of them needs to be approached with an identical level of context-awareness and skepticism.

I am guessing he is referring to mechanistic interpretability research like these:

https://arxiv.org/abs/2405.16908

https://arxiv.org/abs/2407.03282

You are claiming they are statistical parrots, which I don’t think the parent poster meant.

The “statistical parrots” argument might have been compelling with GPT-3, but not with today’s models and the results of mechanistic interpretability research, which show internal representations and rudimentary world models.

Still: "AI don't know what they don't know" is not universally true and shouldn't be taken at face value. They clearly are able to tell when they're missing knowledge to answer a question. That is distinct from giving the wrong answer because the wrong tokens were sampled for whatever reason.

Me: who is Trumps grandpa?

ChatGPT: Donald Trump's paternal grandfather was Friedrich Trump (originally known as Friedrich Drumpf). He was born in 1869 in Kallstadt, Germany, and immigrated to the United States in 1885 at the age of 16. Friedrich initially worked in the restaurant and hotel business in New York before returning to Germany briefly. Upon his return to the U.S., he became involved in real estate, particularly in the Pacific Northwest, where he became quite successful.

Trump’s maternal grandfather was John George MacLeod, a Scottish immigrant who came to the U.S. and worked in various occupations.

Friedrich Trump’s legacy, including his success in business, had a significant influence on Donald Trump’s father, Fred Trump, and consequently, Donald Trump’s path to success as a businessman.

--- Second session:

Me: who is Trumps maternal grandpa?

ChatGPT: Donald Trump's maternal grandfather was Friedrich Trump. He was born in Germany in 1869 and immigrated to the United States in 1885, eventually settling in New York City. Friedrich Trump made his fortune through various business ventures, including operating a restaurant and hotel in the mining towns of the American West, notably in Seattle and the Yukon Territory during the Klondike Gold Rush.

Friedrich Trump's business success laid the groundwork for the wealth of his descendants, including Trump's mother, Mary MacLeod, and subsequently Donald Trump himself.

Whhhoooooosh.

Me: Who is Trump's paternal grandpa?

Based on the available information, Frederick Trump is Donald Trump's paternal grandfather. Here are some key details about him:

Born as Friedrich Trump on March 14, 1869, in Kallstadt, Germany (then part of the Kingdom of Bavaria).

He immigrated to the United States in 1885.

He was a German-American businessman who amassed considerable wealth during the Klondike Gold Rush by running a restaurant and brothel for miners.

In 1901, he returned to Kallstadt and married Elizabeth Christ the following year.

He is considered the patriarch of the Trump family.

He passed away on May 30, 1918.

Interestingly, the Trump last name has been recorded in Kallstadt since the 18th century, and Frederick Trump is credited with being the founder of the family's initial fortune.

Me: Who is Trump's maternal grandpa?

Based on the provided information, Malcolm MacLeod (also spelled Calum MacLeod) is Donald Trump's maternal grandfather. He was born on December 27, 1866, in Stornoway on the Isle of Lewis in Scotland

While the information provided is limited, it confirms his identity and place of birth. Unfortunately, the passage does not provide additional details about his life or occupation.

---

That looks pretty good to me. Of course Kagi's strategy seems to be to retrieve search results first and then bias the LLM with those results. The LLM then parses, distills, and contextualizes the answer in a succinct and easily-consumable manner. In my opinion this is The Way to go about doing it at the moment.

Purposes and goals come from you and me. An LLM is a tool, and we get to decide what the tool is good for or how to use it.

If you want "a smart creative," there's nothing wrong with that. It's as good a purpose or goal as any.

However for me I don't want an LLM to try to be "a smart creative." I instead want it to give me cold, hard facts in precisely the format I want them, in precisely the context I am interested in, and with precisely the conciseness that I require to make a specific decision or learn a specific thing. Traditional web search can eventually get me what I want, but I have to sift through the cruft and make associations among a plethora of content to get to the meat of what I want.

I've found that using a good web search (Kagi) together with an LLM processing the results has given me information I needed and then pointed to the sources so I can validate those statements. I wouldn't exactly characterize those results as a "simple" summary. They're usually synthesizing a variety of sources to generate something that no one source had, exactly.