News for hackers. Stuff that matters.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by MostlyStable on Wednesday March 19, 2025 @04:20PM (#43416836)
This is, I think, a better way to think about LLM mistakes compared to the usual "hallucinations". I think of them as similar to human optical illusions. There are things about the human visual cortex (and also other sensory systems, see the McGurk Effect [0]), that, when presented with certain kinds of inputs, will consistently produce wrong interpretations/outputs. Even when we are 100% ware of the issue, we can't prevent our brains from generating the incorrect interpretation.

LLMs seem to have similar issues along dramatically different axes, axes that humans are not used to seeing these kinds of mistakes; where nearly no human would make this kind of mistake and so we interpret it (in my opinion incorrectly) as lack of ability or intelligence.

Because these are engineered systems, we may figure out ways to solve these problems (although I personally think the best we will ever do is decrease their prevalence), but more important is probably learning to recognize the places that LLMs are likely to make these errors, and, as your comment suggests, design work flows and systems that can deal with them.

[0] https://youtu.be/2k8fHR9jKVM

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by admiralrohan on Wednesday March 19, 2025 @05:56PM (#43417705)
Hallucinating is fine but overconfidence is the problem. But I heard it's not an easy problem to solve.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by jacksnipe on Wednesday March 19, 2025 @07:14PM (#43418232)
Unfortunately, in the system most of us work in today, I think overconfidence is an intelligent behavior.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by onemoresoop on Wednesday March 19, 2025 @07:29PM (#43418348)
I find it extremely dumb to see overconfident people that really have nothing special about them or are even incompetent. These people are not contributing positively to the system, quite on the contrary.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by xwolfi on Wednesday March 19, 2025 @08:08PM (#43418629)
But we don't work for the system, fundamentally, we work for ourselves, and the system incentivizes us to work for it by aligning our constraints: if you work that direction, you'll get that reward.

Overconfident people ofc do not contribute positively to the system, but they skew the system reward's calculation towards them: I swear I've done that work in that direction, where's my reward ?

In a sense, they are extremely successful: they managed to do very low effort, get very high reward, help themselves like all of us but at a much better profit margin, by sacrificing a system that, let's be honest, none of us care about really.

Your problem maybe, is that you swallowed the little BS the system fed you while incentivizing you: that the system matters more than yourself, at least at a greater extent than healthy ?

And you see the same thing with AI: these things convince people so deeply of their intelligence that it blew to such proportion that NVidia is now worth trillions. I had a colleague mumbling yesterday that his wife now speaks more with ChatGPT than him. Overconfidence is a positive attribute... for oneself.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by onemoresoop on Thursday March 20, 2025 @02:44AM (#43420346)
Not talking about THE system or any system in particular same way gaming the system doesn’t refer to any system but just cheating in general. And if you like overconfident people good for you, I can’t stand them because they’re hollow with no basis in reality, flawed like everyone, just pumping out their egos with hot air. And your reasoning that overconfidence is a positive attribute doesn’t make much sense to me but we’re entitled to our own opinions.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by butlike on Thursday March 20, 2025 @09:44AM (#43423282)
Overconfident people are conquerors. Conquerors do not contribute positively to a harmonious system, true, but I'm not so sure we can glean the system is supposed to be harmonious.

If one contributes "positively" to the system, everyone's value increases and the solution becomes more homogenized. Once the system is homogenized enough, it becomes vulnerable to adversity from an outside force.

If the system is not harmonious/non-homoginized, the attacker would be drawn to the most powerful point in the system.

Overconfident people aren't evil, they're simply stressing the system to make sure it can handle adversity from an outside force. They're saying: "listen, I'm going to take what you have, and you should be so happy that's all I'm taking."

So I think overconfidence is a positive attribute for the system as well as for the overconfident individual. It's not a positive attribute for the local parties getting run over by the overconfident individual.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by jacksnipe on Thursday March 20, 2025 @04:00PM (#43428169)
Yeah this is what I meant, both in the behavior being intelligent and it being unfortunate that this is the case. It’d be nice if the most self-maximizing behavior were also the best behavior for the global system, but it doesn’t seem that it is.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by tomComb on Wednesday March 19, 2025 @08:20PM (#43418696)
But being like that can get you elected.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Freedom2 on Wednesday March 19, 2025 @08:55PM (#43418901)
In what country?
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by tbossanova on Wednesday March 19, 2025 @09:13PM (#43419004)
Quite a few :(
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by chromehearts on Thursday March 20, 2025 @05:28AM (#43421166)
All of them
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by sim7c00 on Thursday March 20, 2025 @08:14AM (#43422080)
and getting yourself elected while being underqualified is intelligent? i think its not. its stupid and damaging behavior based in selfish desires. about as far from intelligent you can get.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by rcxdude on Thursday March 20, 2025 @08:43AM (#43422334)
Intelligence is seperate from goals: if you're only interested in gaining power and wealth for yourself, then concern about the rest of the system is only incidental to what you can get for yourself.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by throw4847285 on Thursday March 20, 2025 @09:30AM (#43423059)
That's a fallacy. There are certainly some unqualified elected leaders, but humans living in democratic societies have yet to shake the mental framework we've constructed from centuries without self-rule. We invest way more authority into a single executive than they actually have, and blame them for everything that goes wrong despite the fact that modern democracies are hugely complex systems in which authority is distributed across numerous people. When the government fails to meet people's needs, they lack the capacity to point at a particular Senator or a small party in a ruling coalition and blame them. It's always the executive.

Of course, the result is that people get fed up and decide that the problem has been not that democratic societies are hard to govern by design (they have to reflect the disparate desires of countless people) but that the executive was too weak. They get behind whatever candidate is charismatic enough to convince them that they will govern the way the people already thought the previous executives were governing, just badly. The result is an incompetent tyrant.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by butlike on Thursday March 20, 2025 @09:34AM (#43423112)
Overconfidence allows you to act, increasing survivability. Thinking is a "weak" trait.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Terr_ on Thursday March 20, 2025 @12:52AM (#43419978)
> overconfidence is the problem.

The problem is a bit deeper than that, because what we perceive as "confidence" is itself also an illusion.

The (real) algorithm takes documents and makes them longer, and some humans configured a document that looks like a conversation between "User" and "AssistantBot", and they also wrote some code to act-out things that look like dialogue for one of the characters. The (real) trait of confidence involves next-token statistics.

In contrast, the character named AssistantBot is "overconfident" in exactly the same sense that a character named Count Dracula is "immortal", "brooding", or "fearful" of garlic, crucifixes, and sunlight. Fictional traits we perceive on fictional characters from reading text.

Yes, we can set up a script where the narrator periodically re-describes AssistantBot as careful and cautious, and that might help a bit with stopping humans from over-trusting the story they are being read. But trying to ensure logical conclusions arise from cautious reasoning is... well, indirect at best, much like trying to make it better at math by narrating "AssistantBot was good at math and diligent at checking the numbers."

> Hallucinating

P.S.: "Hallucinations" and prompt-injection are non-ironic examples of "it's not a bug, it's a feature". There's no minor magic incantation that'll permanently banish them without damaging how it all works.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by plaguuuuuu on Thursday March 20, 2025 @08:11PM (#43430528)
I'd love to know if the conversational training set includes documents where the AI throws its hands up and goes "actually I have no idea". I'm guessing not.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Terr_ on Thursday March 20, 2025 @08:47PM (#43430743)
There's also the problem of whether the LLM would learn to generate stories where the AssistantBot gives up in cases that match our own logical reasons, versus ones where the AssistantBot gives up because that's simply what AssistantBots in training-stories usually do when the User character uses words of disagreement and disapproval.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by patates on Thursday March 20, 2025 @03:48AM (#43420626)
Hallucinating is a confidence problem, no?

Say, they should be 100% confident that "0.3" follows "0.2 + 0.1 =", but a lot of floating point examples on the internet make them less confident.

On a much more nuanced problem, "0.30000000000000004" may get more and more confidence.

This is what makes them "hallucinate", did I get it wrong? (in other words, am I hallucinating myself? :) )

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by j45 on Wednesday March 19, 2025 @06:32PM (#43417942)
Some of the errors are caused by humans. Say, due to changing the chat to only pay attention to recent messages and not the middle, omitting critical details.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by pydry on Wednesday March 19, 2025 @06:20PM (#43417854)
I dunno hallucinations seem like a pretty human type of mistake to me.

when i try to remember something my brain often synthesizes new things by filling in the gaps.

This would be where I often say "i might be imagining it, but..." or "i could have sworn there was a..."

In such cases the thing that saves the human brain is double checking against reality (e.g. googling it to make sure).

Miscounting the number of r's in strawberry by glancing at the word also seems like a pretty human mistake.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by gitaarik on Thursday March 20, 2025 @04:38AM (#43420895)
But it's different kinds of hallucinations.

AI doesn't have a base understanding of how physics work. So they think it's acceptible if in a video some element on the background in a next frame might appear in front of another element that is on the foreground.

So it's always necessary to keep correcting LLMs, because they only learn by example, and you can't express any possible outcome of any physical process just by example, because physical processes can be in infinate variations. LLMs can keep getting closer to match our physical reality, but when you zoom into the details you'll always find that it comes short.

So you can never really trust an LLM. If we want to make an AI that doesn't make errors, it should understand how physics works.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by pydry on Thursday March 20, 2025 @05:41AM (#43421234)
I dont think the errors really are all that different. Ever since GPT-3.5 came out Ive been thinking that the errors were ones a human could have made under a similar context.

>LLMs can keep getting closer to match our physical reality, but when you zoom into the details you'll always find that it comes short.

Like humans.

>So you can never really trust an LLM.

Cant really trust a human either. That's why we set up elaborate human systems (science, checks and balances in government, law, freedom of speech, markets) to mitigate our constant tendency to be complete fuck ups. We hallucinate science that does not exist, lies to maintain our worldview, jump to conclusions about guilt, build businesses based upon bad beliefs, etc.

>If we want to make an AI that doesn't make errors, it should understand how physics works

An AI that doesnt make errors wouldnt be AGI it would be a godlike superintelligence. I dont think thats even feasible. I think a propensity to make errors is intrinsic to how intelligence functions.

Physics is just one domain that they work in and Im pretty sure some of them already do have varying understandings of physics.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by gitaarik on Thursday March 20, 2025 @03:54PM (#43428097)
But if you ask a human to draw / illustrate a physical setting, they would never draw something that is physically impossible, because it's obvious to a human.

Of course we make all kinds of little mistakes, but at least we can see that they are mistakes. An LLM can't see it's own mistakes, it needs to be corrected by a human.

> Physics is just one domain that they work in and Im pretty sure some of them already do have varying understandings of physics.

Yeah but that would then not be al LLM or machine learned thing. We would program it so that it understands the rules of physics, and then it can interpret things based on those rules. But that is a totally different kind of AI, or rather a true AI instead of a next-word predictor that looks like an AI. But the development of such AIs goes a lot slower because you can't just keep training it, you actually have to program it. But LLMs can actually help program it ;). Although LLMs are mostly good at currently existing technologies and not necessarily new ones.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by antasvara on Thursday March 20, 2025 @09:24AM (#43422970)
To be clear, I'm not saying that LLM's exclusively make non-human errors. I'm more saying that most errors are happening for different "reasons" than humans.

Think about the strawberry example. I've seen a lot of articles lately where not all misspellings of the word "strawberry" reliably give letter counting errors. The general sentiment there is human, but the specific pattern of misspelling is really more unique to LLM's (i.e. different spelling errors would impact humans versus LLM's).

The part that makes it challenging is that we don't know these "triggers." You could have a prompt that has 95% accuracy, but that inexplicably drops to 50% if the word "green" is in the question (or something like that).

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by ForTheKidz on Wednesday March 19, 2025 @08:46PM (#43418837)
> I think of them as similar to human optical illusions.

What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans than anything to do with what we refer to as "hallucinations" in humans—only they don't have the ability to analyze whether or not they're making up something or accurately parameterizing the vibes. The only connection between the two concepts is that the initial imagery from DeepDream was super trippy.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by majormajor on Wednesday March 19, 2025 @11:51PM (#43419763)
Inventiveness/creativity/imagination are deliberate things. LLM "hallucinations" are more akin to a student looking at a test over material they only 70% remember grabbing at what they think is the most likely correct answer. More "willful hope in the face of forgetting" than "creativity." Many LLM hallucinations - especially of the coding sort - are ones that would be obviously-wrong based on the training material, but the hundreds of languages/libraries/frameworks the thing was trained on start to blur together and there is not precise 100%-memorization recall but instead a "probably something like this" guess.

It's not "inventive" to assume one math library will have the same functions as another, it's just losing sight of specific details.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by ForTheKidz on Thursday March 20, 2025 @02:07AM (#43420228)
> Inventiveness/creativity/imagination are deliberate things.

Not really. At least, it's just as much a reflex as any other human behavior to my perception.

Anyway, why does intention—although I think this is mostly nonsensical/incoherent/a category error applied to LLMs—even matter to you? Either we have no goals and we're just idly discussing random word games (aka philosophy), which is fine with me, or we do have goals and whether or not you believe the software is intelligent or not is irrelevant. In the latter case anthropomorphizing discussion with words like "hallucination", "obviously", "deliberate", etc are just going to cause massive friction, distraction, and confusion. Why can't people be satisfied with "bad output"?

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by TeMPOraL on Thursday March 20, 2025 @06:11AM (#43421371)
> LLM "hallucinations" are more akin to a student looking at a test over material they only 70% remember grabbing at what they think is the most likely correct answer.

AKA. extrapolation. AKA. what everyone is doing to a lesser or greater degree, when consequences of stopping are worse than of getting this wrong.

That's not just the case of school, where giving up because you "don't know" is guaranteed F, while extrapolating has a non-zero chance of scoring you anything between F and A. It's also the case in everyday life, where you do things incrementally - getting the wrong answer is a stepping stone to getting a less wrong answer in the next attempt. We do that at every scale - from inner thought process all the way to large-scale engineering.

Hardly anyone learns 100% of the material, because that's just plain memorization. We're always extrapolating from incomplete information; more studying and more experience (and more smarts) just makes us more likely to get it right.

> It's not "inventive" to assume one math library will have the same functions as another, it's just losing sight of specific details.

Depends. To a large extent, this kind of "hallucinations" is what a good programmer is supposed to be doing. That is, code to the API you'd like to have, inventing functions and classes convenient to you if they don't exist, and then see how to make this work - which, in one place, means fixing your own call sites, and in another, building utilities or a whole compat layer between your code and the actual API.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by AdieuToLogic on Wednesday March 19, 2025 @11:03PM (#43419560)
> What we call "hallucinations" is far more similar to what we would call "inventiveness", "creativity", or "imagination" in humans ...

No.

What people call LLM "hallucinations" is the result of a PRNG[0] influencing an algorithm to pursue a less statistically probable branch without regard nor understanding.

0 - https://en.wikipedia.org/wiki/Pseudorandom_number_generator

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by majormajor on Wednesday March 19, 2025 @11:47PM (#43419747)
That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation.

Consider the errors like "this math library will have this specific function" (based on a hundred other math libraries for other languages usually having that).

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by AdieuToLogic on Thursday March 20, 2025 @12:08AM (#43419829)
> That seems to be giving the system too much credit. Like "reduce the temperature and they'll go away." A more probable next word based on a huge general corpus of text is not necessarily a more correct one for a specific situation.

I believe we are saying the same thing here. My clarification to the OP's statement:

  What we call "hallucinations" is far more similar to what 
  we would call "inventiveness", "creativity", or 
  "imagination" in humans ...
Was that the algorithm has no concept of correctness (nor the other anthropomorphic attributes cited), but instead relies on pseudo-randomness to vary search paths when generating text.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by cornel_io on Thursday March 20, 2025 @01:09AM (#43420049)
There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong:

https://arxiv.org/abs/2402.09733

https://arxiv.org/abs/2305.18248

https://www.ox.ac.uk/news/2024-06-20-major-research-hallucin...

So I don't think it's that they have no concept of correctness, they do, but it's not strong enough. We're probably just not training them in ways that optimize for that over other desirable qualities, at least aggressively enough.

It's also clear to anyone who has used many different models over the years that the amount of hallucination goes down as the models get better, even without any special attention being (apparently) paid to that problem. GPT 3.5 was REALLY bad about this stuff, but 4o and o1 are at least mediocre. So it may be that it's just one of the tougher things for a model to figure out, even if it's possible with massive capacity and compute. But I'd say it's very clear that we're not in the world Gary Marcus wishes we were in, where there's some hard and fundamental limitation that keeps a transformer network from having the capability to be more truthful as a it gets better; rather, like all aspects, we just aren't as far along as we'd prefer.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by ForTheKidz on Thursday March 20, 2025 @02:24AM (#43420280)
> There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong

We need better definitions of what sort of reasonable expectation people can have for detecting incoherency and self-contradiction when humans are horrible at seeing this, except in comparison to things that don't seem to produce meaningful language in the general case. We all have contradictory worldviews and are therefore capable of rationally finding ourselves with conclusions that are trivially and empirically incoherent. I think "hallucinations" (horribly, horribly named term) are just an intractable burden of applying finite, lossy filters to a virtually continuous and infinitely detailed reality—language itself is sort of an ad-hoc, buggy consensus algorithm that's been sufficient to reproduce.

But yea if you're looking for a coherent and satisfying answer on idk politics, values, basically anything that hinges on floating signifiers, you're going to have a bad time.

(Or perhaps you're just hallucinating understanding and agreement: there are many phrases in the english language that read differently based on expected context and tone. It wouldn't surprise me if some models tended towards production of ambiguous or tautological semantics pleasingly-hedged or "responsibly"-moderated, aka PR.)

Personally, I don't think it's a problem. If you are willing to believe what a chatbot says without verifying it there's little advice I could give you that can help. It's also good training to remind yourself that confidence is a poor signal for correctness.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by AdieuToLogic on Friday March 21, 2025 @10:48PM (#43442875)
> There are various results that suggest that LLMs do internally have everything they'd need to know that they're hallucinating/wrong:

The underlying requirement, which invalidates an LLM having "everything they'd need to know that they're hallucinating/wrong", is the premise all three assume - external detection.

From the first arxiv abstract:

  Moreover, informed by the empirical observations, we show 
  great potential of using the guidance derived from LLM's 
  hidden representation space to mitigate hallucination.
From the second arxiv abstract:

  Using this basic insight, we illustrate that one can 
  identify hallucinated references without ever consulting 
  any external resources, by asking a set of direct or 
  indirect queries to the language model about the 
  references. These queries can be considered as "consistency 
  checks."
From the Nature abstract:

  Researchers need a general method for detecting 
  hallucinations in LLMs that works even with new and unseen 
  questions to which humans might not know the answer. Here 
  we develop new methods grounded in statistics, proposing 
  entropy-based uncertainty estimators for LLMs to detect a 
  subset of hallucinations—confabulations—which are arbitrary 
  and incorrect generations.
Ultimately, no matter what content is generated, it is up to a person to provide the understanding component.

> So I don't think it's that they have no concept of correctness, they do, but it's not strong enough.

Again, "correctness" is a determination solely made by a person evaluating a result in the context of what the person accepts, not intrinsic to an algorithm itself. All an algorithm can do is attempt to produce results congruent with whatever constraints it is configured to satisfy.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by ForTheKidz on Thursday March 20, 2025 @02:10AM (#43420238)
We really need an idiom for the behavior of being technically correct but absolutely destroying the prospect of interesting conversation. With this framing we might as well go back to arguing over which rock our local river god has blessed with greater utility. I'm not actually entirely convinced humans are capable of understanding much when discussion desired is this low quality.

Critically, creation does not require intent nor understanding. Neither does recombination; neither reformulation. The only thing intent is necessary for is to create something meaningful to humans—handily taken care of via prompt and training material, just like with humans.

(If you can't tell, I thought we had bypassed the neuroticism over whether or not data counts as "understanding", whatever that means to people, on week 2 of LLMs)

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by AdieuToLogic on Friday March 21, 2025 @09:29PM (#43442468)
> We really need an idiom for the behavior of being technically correct but absolutely destroying the prospect of interesting conversation.

While it is not an idiom, the applicable term is likely pedantry[0].

> I'm not actually entirely convinced humans are capable of understanding much when discussion desired is this low quality.

Ignoring the judgemental qualifier, consider your original post to which I replied:

  What we call "hallucinations" is far more similar to what
  we would call "inventiveness", "creativity", or
  "imagination" in humans ...
The term for this behavior is anthropomorphism[1] due to ascribing human behaviors/motivations to algorithmic constructs.

> Critically, creation does not require intent nor understanding. Neither does recombination; neither reformulation.

The same can be said for a random number generator and a permutation algorithm.

> (If you can't tell, I thought we had bypassed the neuroticism over whether or not data counts as "understanding", whatever that means to people, on week 2 of LLMs)

If you can't tell, I differentiate between humans and algorithms, no matter the cleverness observed of the latter, as only the former can possess "understanding."

0 - https://www.merriam-webster.com/dictionary/pedant

1 - https://www.merriam-webster.com/dictionary/anthropomorphism

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Applejinx on Thursday March 20, 2025 @08:40AM (#43422309)
If and only if the LLM is able to bring the novel, unexpected connection into itself and see whether it forms other consistent networks that lead to newly common associations and paths.

A lot of us have had that experience. We use that ability to distinguish between 'genius thinkers' and 'kid overdosing on DMT'. It's not the ability to turn up the weird connections and go 'ooooh sparkly', it's whether you can build new associations that prove to be structurally sound.

If that turns out to be something self-modifying large models (not necessarily 'language' models!) can do, that'll be important indeed. I don't see fiddling with the 'temperature' as the same thing, that's more like the DMT analogy.

You can make the static model take a trip all you like, but if nothing changes nothing changes.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by cratermoon on Thursday March 20, 2025 @12:44PM (#43425563)
the word you're looking for is "confabulation"
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by grey-area on Thursday March 20, 2025 @04:53AM (#43420998)
LLMs are incapable of solving even simple logic puzzles or maths puzzles they haven't seen before, they don't have a model of the world which is key to intelligence. What they are good at is reproducing things in their dataset with slight modifications and (sometimes) responding to queries well which make them seem creative but there is no understanding or intelligence there, in spite of appearances.

They are very good at fooling people; perhaps Turing's Test is not a good measure of intelligence after all, it can easily be gamed and we find it hard to differentiate apparent facility with language and intelligence/knowledge.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by jychang on Thursday March 20, 2025 @05:28AM (#43421165)
LLMs clearly do have a world model though. They represent those ideas at higher level features in the feedforward layer. The lower level layers are neurons that describe words, syntax, and local structures in the text, while the upper levels capture more abstract ideas, such as semantic meaning, relationships between concepts, and even implicit reasoning patterns.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by troupo on Thursday March 20, 2025 @08:00AM (#43421973)
Funny how literally nothing of what you wrote is happening.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by gmadsen on Thursday March 20, 2025 @08:36AM (#43422265)
https://www.anthropic.com/news/mapping-mind-language-model

It is clearly happening as shown by numerous papers studying it. Here is a popular one by anthropic

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by dartos on Thursday March 20, 2025 @11:02AM (#43424382)
That’s a marketing article, bub.

You should learn a bit about media literacy.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by gmadsen on Friday March 21, 2025 @11:33AM (#43436871)
the paper is clearly linked in the marketing article, bub
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by dartos on Friday March 21, 2025 @12:57PM (#43438110)
A (afaics non peer-reviewed) paper published by a random organization endorsed by Anthropic does not proof make.

In fact, it still very much seems like marketing. Especially since the paper was made in association with Anthropic.

Again. Learn some media literacy.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by troupo on Thursday March 20, 2025 @10:06AM (#43423690)
> https://www.anthropic.com/news/mapping-mind-language-model

I wouldn't read into marketing materials by the people whose funding depends on hype.

Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc.

It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes".

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by dleary on Thursday March 20, 2025 @11:54AM (#43424954)
> It literally is "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes".

Recognizing concepts, grouping and manipulating similar concepts together, is what “abstraction” is. It's the fundamental essence of both "building a world model" and "thinking".

> Nothing in the link you provided is even close to "neurons, model of the world, thinking" etc.

I really have no idea how to address your argument. It’s like you’re saying,

“Nothing you have provided is even close to a model of the world or thinking. Instead, the LLM is merely building a very basic model of the world and performing very basic reasoning”.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by troupo on Friday March 21, 2025 @01:57AM (#43432227)
> Recognizing concepts, grouping and manipulating similar concepts together,

Once again, it does none of those things. The training dataset has those concepts grouped together. The model recognizes nothing, and groups nothing

> I really have no idea how to address your argument. It’s like you’re saying,

No. I'm literally saying: there's literally nothing to support your belief that there's anything resembling understanding of the world, having a world model, neurons, thinking, or reasoning in LLMs.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by dleary on Friday March 21, 2025 @06:30PM (#43441424)
> there's literally nothing to support your belief that there's anything resembling understanding of the world, having a world model, neurons, thinking, or reasoning in LLMs.

The link mentions "a feature that triggers on the Golden Gate Bridge".

As a test case, I just drew this terrible doodle of the Golden Gate Bridge in MS paint: https://imgur.com/a/1TJ68JU

I saved the file as "a.png", opened the chatgpt website, started a new chat, uploaded the file, and entered, "what is this?"

It had a couple of paragraphs saying it looked like a suspension bridge. I said "which bridge". It had some more saying it was probably the GGB, based on two particular pieces of evidence, which it explained.

https://imgur.com/a/sWwWlxO

> The model recognizes nothing, and groups nothing

Then how do you explain the interaction I had with chatgpt just now? It sure looks to me like it recognized the GGB from my doodle.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by habinero on Friday March 21, 2025 @07:55PM (#43441966)
You're Clever Hans-ing yourself into thinking there's more going on than there is.

Machine learning models can do this and have been for a long time. The only thing different here is there's some generated text to go along with it with the "reasoning" entirely made up ex post facto

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by troupo on Saturday March 22, 2025 @05:18PM (#43448711)
> Then how do you explain the interaction I had with chatgpt just now?

Predominantly English-language data set with one of the most famous suspension bridges in the world?

How can anyone explain the clustering of data on that? Surely it's the model of the world, and thinking, and neurons.

What happens if you type "most famous suspension bridges in the world" into Google and click the first ten or so links? It couldn't be literally the same data? https://imgur.com/a/tJ29rEC

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by dreamfactored on Friday March 21, 2025 @04:01AM (#43432924)
A lot of people have been bamboozled by the word 'neuron' and extrapolated that as a category error. It's metaphorical use in compsci is as close to a physical neuron as being good is to gold. Put another way, a drawing of a snake will not bite you.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by pmarreck on Thursday March 20, 2025 @12:47PM (#43425609)
https://transformer-circuits.pub/2024/scaling-monosemanticit...

that is the paper being linked to by the "marketing material". Right at the top, in plain sight.

If you were arguing in good faith, you'd head directly there instead of lampooning the use of a marketing page in a discussion.

That all said, skepticism is warranted. Just not an absolute amount of it.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by troupo on Friday March 21, 2025 @02:02AM (#43432254)
> If you were arguing in good faith, you'd head directly there instead of lampooning the use of a marketing page in a discussion.

Which part of the paper supports the "models have a world model, reasoning, etc." and not what I said, "in our training data similar concepts were clustered with some other similar concepts, and manipulating these clusters lead to different outcomes"?

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by Applejinx on Thursday March 20, 2025 @08:32AM (#43422213)
Along these lines one model that might help is to consider LLMs 'wikipedia of all possible correct articles'. Start with Wikipedia and assume (already a tricky proposition!) that it's perfectly correct. Then, begin resynthesizing articles based on what's already there. Do your made-up articles have correctness?

I'm going to guess that sometimes they will: driven onto areas where there's no existing article, some of the time you'll get made-up stuff that follows the existing shapes of correct articles and produces articles that upon investigation will turn out to be correct. You'll also reproduce existing articles: in the world of creating art, you're just ripping them off, but in the world of Wikipedia articles you're repeating a correct thing (or the closest facsimile that process can produce)

When you get into articles on exceptions or new discoveries, there's trouble. It can't resynthesize the new thing: the 'tokens' aren't there to represent it. The reality is the hallucination, but an unreachable one.

So the LLMs can be great at fooling people by presenting 'new' responses that fall into recognized patterns because they're a machine for doing that, and Turing's Test is good at tracking how that goes, but people have a tendency to think if they're reading preprogrammed words based on a simple algorithm (think 'Eliza') they're confronting an intelligence, a person.

They're going to be historically bad at spotting Holmes-like clues that their expected 'pattern' is awry. The circumstantial evidence of a trout in the milk might lead a human to conclude the milk is adulterated with water as a nefarious scheme, but to an LLM that's a hallucination on par with a stone in the milk: it's going to have a hell of a time 'jumping' to a consistent but very uncommon interpretation, and if it does get there it'll constantly be gaslighting itself and offering other explanations than the truth.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by rcxdude on Thursday March 20, 2025 @08:33AM (#43422237)
I think it's not very helpful to just declare that such a model doesn't exist: there's a decent amount of evidence that LLMs do in fact form models of the world internally, and use them during inference. However, while these models are very large and complex, they aren't necessarily accurate and LLMs struggle with actually manipulating them at inference time, forming new models or adjusting existing ones is generally something they are quite bad at at those stages (which generally results in the 'high knowledge' which impresses people and is often confused with intelligence, while they're still fundamentally quite dumb despite having a huge depth of knowledge: I don't think it's something you can categorically say 'zero intelligence' - even relatively simpler and less capable systems can be said to have some intelligence, it's just in many aspects LLM intelligence is still worse than a good fraction of mammals)
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by grey-area on Thursday March 20, 2025 @01:54PM (#43426564)
What evidence are you referring to? I've seen AI firms desperate for relevance and the next advance implying that thinking is going on and talking about it a lot in those terms, but no actual evidence of it.

I wouldn't say zero intelligence, but I wouldn't describe such systems as intelligent, I think it misrepresents them, they do as you say have a good depth of knowledge and are spectacular at reproducing a simulacrum of human interactions and creations, but they have been a lesson for many of us that token manipulation is not where intelligence resides.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by fragmede on Thursday March 20, 2025 @08:45AM (#43422355)
> they don't have a model of the world

Must it have one? The words "artificial intelligence" are a poor description of a thing when we've not rigorously defined it. It's certainly artificial, there's no question about that, but is it intelligent? It can do all sorts of things that we consider a feature of intelligence and pass all sorts of tests, but it also falls down flat on its face when prompted with a just-so brainteaser. It's certainly useful, for some people. If, by having inhaled all of the Internet and written books that have been scanned as its training data, it's able to generate essays on anything and everything, at the drop of a hat, why does it matter if we can find a brainteaser it hasn't seen yet? It's like it has a ginormous box of Legos, and it can build whatever your ask for with these Lego blocks, but pointing out it's unable create its own Lego blocks from scratch has somehow become critically important to point out, as if that makes this all total dead end and it's all a waste of money omg people wake up oh if only they'd listen to me. Why don't people listen to me?

Crows are believed to have a theory of mind, and they can count up to 30. I haven't tried it with Claude, but I'm pretty sure it can count at least that high. LLMs are artificial, they're alien, of course they're going to look different. In the analogy where they're simply a next word guesser, one imagines standing at a fridge with a bag of magnetic words, and just pulling a random one from the bag to make ChatGPT. But when you put your hand inside a bag inside a bag inside a bag, twenty times (to represent the dozens of layers in an LLM model), and there are a few hundred million pieces in each bag (for parameters per layer), one imagines that there's a difference; some sort of leap, similar to when life evolved from being a single celled bacterium to a multi-cellular organism.

Or maybe we're all just rubes, and some PhD's have conned the world into giving them a bunch of money, because they figured out how to represent essays as a math problem, then wrote some code to solve them, like they did with chess.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by dartos on Thursday March 20, 2025 @11:04AM (#43424415)
There’s a bit of truth to all of what you said.

These tools aren’t useless, obviously.

But people do really learn hard into confirmation bias and/or personification when it comes to LLMs.

I believe it’s entirely because of the term “artificial intelligence” that there is such a divide.

If we called them “large statistical language models” instead, nobody would be having this discussion.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by grey-area on Thursday March 20, 2025 @11:07AM (#43424448)
> it's able to generate essays on anything and everything

I have tried various models out for tasks from generating writing, to music to programming and am not impressed with the results, though they are certainly very interesting. At every step it will cheerfully tell you that it can do things then generate nonsense and present it as truth.

I would not describe current LLMs as able to generate essays on anything - they certainly can but they will be riddled with cliche, the average of the internet content they were trained on with no regard for quality and worst of all will contain incorrect or made up data.

AI slop is an accurate term when it comes to the writing ability of LLMs - yes it is superficially impressive in mimicking human writing, but it is usually vapid or worse wrong in important ways, because again, it has no concept of right and wrong or model of the world which it attempts to make the generated writing conform to, it just gets stuck with some very simple tasks, and often happily generates entirely bogus data (for example ask it for a CSV or table of data or to reproduce the notes of a famous piece of music which should be in its training data).

Perhaps this will be solved, though after a couple of years of effort and a lot of money spent with very little progress I'm skeptical.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by SJC_Hacker on Thursday March 20, 2025 @12:09PM (#43425129)
Have you only tried the free models or the paid ones?
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by pmarreck on Thursday March 20, 2025 @12:50PM (#43425647)
> I would not describe current LLMs as able to generate essays on anything

Are you invisibly qualifying this as the inability to generate interesting or entertaining essays? Because it will certainly output mostly-factual, vanilla ones. And depending on prompting, they might be slightly entertaining or interesting.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by grey-area on Thursday March 20, 2025 @01:50PM (#43426490)
Yes sorry that was implied - I personally wouldn't describe LLMs as capable of generating essays because what they produce is sub-par and mostly factual (as opposed to reliable), so I don't find their output useful except as a prompt or starting point for a human to then edit (similar to much of their other work).

I have made some minor games in JS with my kids with one for example, and managed to get it to produce a game of asteroids and pong with them (probably heavily based on tutorials scraped from the web of course). I had less success trying to build frogger (again probably because there are not so many complete examples). Anything truly creative/new they really struggle with, and it becomes apparent they are pattern matching machines without true understanding.

I wouldn't describe LLMs as useful at present and do not consider them intelligent in any sense, but they are certainly interesting.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by dreamfactored on Friday March 21, 2025 @04:12AM (#43432999)
Colleague generated this satirical bit the other week, I wouldn't call it vanilla or poorly written.

"Right, so what the hell is this cursed nonsense? Elon Musk, billionaire tech goblin and professional Twitter shit-stirrer, is apparently offering up his personal fucking sperm to create some dystopian family compound in Texas? Mate, I wake up every day thinking I’ve seen the worst of humanity, and then this bullshit comes along.

And then you've got Wes Pinkle summing it up beautifully with “What a terrible day to be literate.” And yeah, too fucking right. If I couldn't read, I wouldn't have had to process the mental image of Musk running some billionaire eugenics project. Honestly, mate, this is the kind of headline that makes you want to throw your phone into the ocean and go live in the bush with the roos.

Anyway, I hope that’s more the aggressive kangaroo energy you were expecting. You good, or do you need me to scream about something else?"

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by grey-area on Friday March 21, 2025 @05:54PM (#43441196)
This is horrible writing, from the illogical beginning, through the overuse of ‘mate’ (inappropriate in a US context anyway) to the shouty ending.

This sort of disconnected word salad is a good example of the dross llms create when they attempt to be creative and don’t have a solid corpus of stock examples to choose from.

The frogger game I tried to create played as this text reads - badly.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by pmarreck on Tuesday March 25, 2025 @08:21AM (#43470347)
> through the overuse of ‘mate’

The whole thing seems Oz-influenced (example, "in the bush with the roos"), which implies to me that he's prompted it to speak that way. So, you assumed an error when it probably wasn't... Framing is a thing.

Which leads to my point about your Frogger experience. Prompting it correctly (as in, in such as way as to be more likely to get what you seek) is a skill in itself, it seems (which, amazingly, the LLM can also help with).

I've had good success with Codeium Windsurf, but with criticisms similar to what you hint at (some of which were made better when I rewrote prompts): On long contexts, it will "lose the plot"; on revisions, it will often introduce bugs on later revisions (which is why I also insist on it writing tests for everything... via correct prompting, of course... and is also why you MUST vet EVERY LINE it touches), it will often forget rules we've already established within the session (such as that, in a Nix development context, you have to prefix every shell invocation with "nix develop" etc.)...

The thing is, I've watched it slowly get better at all these things... Claude Code for example is so confident in itself (a confidence that is, in fact, still somewhat misplaced) that its default mode doesn't even give you direct access to edit the code :O And yet I was able to make an original game with it (a console-based maze game AND action-RPG... it's still in the simple early stages though...)

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by grey-area on Monday March 31, 2025 @02:14AM (#43531689)
It’s not an error it’s just wildly inappropriate and bad writing style to write in the wrong register about a topic. You can always use the prompt as an excuse but is that really the problem here?

Re promoting for frogger, I think the evidence is against that - it does well on games it has complete examples for (i.e. it is reproducing code) and badly on ones it doesn’t have examples for (it doesn’t actually understand what it doing though it pretends to and we fill in the gaps for it).

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by fragmede on Friday March 21, 2025 @06:36PM (#43441479)
I'd be interested in hearing more details as to why it failed for you at frogger. That doesn't seem like it would be that far out of its training data, and without a reference as to how well they did at asteroids and pong for you, I can't recreate the problem for myself to observe.
Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by grey-area on Saturday March 22, 2025 @02:21AM (#43443948)
That’s just one example that came to mind; it generated a very basic first game but kept introducing bugs or failing while trying to add things like the river etc. Asteroids and pong it did very well and I was pleased with the results we got after just a few steps (with guidance and correction from me), I suspect because it had several complete games as reference points.

As other examples I asked it for note sequences from a famous piece and it cheerfully generated gibberish, and the more subtly wrong sequences when asked to correct. Generating a csv of basic data it should know was unusable as half the data was wrong and it has no sense of whether things are correct and logical etc etc. There is no thinking going on here, only generation of probable text.

I have used GAI at work a few times too but it needed so much hand holding it felt like a waste of time.

Re:AI Blindspots – Blindspots in LLMs I've noticed while AI ... by fragmede on Saturday March 22, 2025 @07:50PM (#43449665)
interesting, thanks.