News for hackers. Stuff that matters.
Re:Do large language models need sensory grounding for meani... by abeppu on Sunday March 26, 2023 @08:57PM (#35321578)
I'm on board with a lot of what's in this deck, but I take issue with the argument on slide 9. Roughly, the probability that an LLM-provided answer is fully correct decreases exponentially with the length of the answer. I think that's trivially true, but it's also true for human-provided answers (a full non-fiction book is going to have some errors), so it doesn't really get to the core problem with LLMs specifically.

In much of the rest of the deck, it's just presumed that any variable named x comes from the world in some generic way, which doesn't really distinguish why those are a better basis for knowledge or reasoning than the linguistic inputs to LLMs.

I think we're at the point where people working in these areas need some exposure to the prior work on philosophy of mind and philosophy of language.

Re:Do large language models need sensory grounding for meani... by elefanten on Sunday March 26, 2023 @10:42PM (#35322279)
Appreciate your take, not arguing with it.

But I’d imagine Lecun has more than passing familiarity with those. This deck was put out with the Philosophy dept and he had a panel debate with NYU profs across depts (inc phil) recently on this topic.

I suspect this is all pushing the top Phil Lang and Phil Mind to their limits too. Besides, if those subjects were anywhere near resolved (or even… decently understood), they probably wouldn’t be in the Phil dept any more.

Re:Do large language models need sensory grounding for meani... by Grothendank on Monday March 27, 2023 @12:01AM (#35322720)
I feel like Lecun did a poor job of localizing the technique relative to any philosophy, or really explaining what it does for a nontechnical audience.

Saying he gives talks to philosophers, or saying this pushes philosophy to its limits doesn't fix the problem that lecun does a poor job - in this presentation - of philosophically motivating the proposal.

Perhaps I am wrong, and you can point out exactly how lecun explicates the philosophy in the presentation - perhaps it's really embedded in the maths, which I have not appreciated.

Appealing to lecun's authority won't fix the opacity of the presentation. But interpreting it can help! Are you up for it?

Re:Do large language models need sensory grounding for meani... by elefanten on Monday March 27, 2023 @04:26AM (#35324248)
I haven't even given it a deep read, so I unfortunately can't help shed any light. From my quick read, it didn't seem to me that laying out a digestible or rigorous philosophical perspective was really the point here. It seemed more directly biology inspired than philosophy.

I also don't seek to valorize Lecun. But he was a very early figure working on these technologies and from the beginning there was a neuroscience-inspired impetus to machine learning.

My point was sort of the opposite: that assuming Lecun doesn't clear the bar of having "exposure to the prior work on philosophy of mind and philosophy of language" seems like a weak bet.

edit: For clarity, I'm making the assumption that lifelong AI researchers who put time into learning from neuroscience... would also gravitate towards and seek to learn from the nearest relevant branches of philosophy.

Re:Do large language models need sensory grounding for meani... by gsjbjt on Sunday March 26, 2023 @11:54PM (#35322686)
The point is that LLMs can’t backtrack after deciding on a token. So the probability at least one token along a long generation will lead you down the wrong path does indeed increase as the sequence gets longer (especially since we typically sample from these things), whereas humans can plan their outputs in advance, revise/refine, etc.
Re:Do large language models need sensory grounding for meani... by morkalork on Monday March 27, 2023 @12:13AM (#35322775)
To extend the non-fiction book example, the author doesn't write it in a one-shot forward pass, they go back and edit.
Re:Do large language models need sensory grounding for meani... by wordpad25 on Monday March 27, 2023 @12:21AM (#35322814)
nothing is stopping you from doing the same with LLMs, you can have it go back over it’s response and check it for accuracy
Re:Do large language models need sensory grounding for meani... by andrewprock on Monday March 27, 2023 @02:10AM (#35323423)
What is stopping the LLMs from doing the same?
Re:Do large language models need sensory grounding for meani... by bobbylarrybobby on Monday March 27, 2023 @01:12AM (#35323075)
Do you know of any work on holistic response quality in LLMs? We currently have the LLM equivalent of the html line break and hyphenation algorithm, when what we want is the LaTeX version of that algorithm.
Re:Do large language models need sensory grounding for meani... by abeppu on Monday March 27, 2023 @01:26AM (#35323154)
Humans can backtrack, but the probability of an "correct" output is still (1-epsilon)^n. Not only can any token introduce an error, but the human author will not perfectly catch errors they have previously introduced. The epsilon ought to be lower for humans, but it's not zero.

But more to the point, in the deck provided, Lecun's point is _not_ about backtracking per se. The highlighted / red text on the preceding slide is:

> LLMs have no knowledge of the underlying reality > They have no common sense & they can't plan their answer

Now, we generally generate from LLMs by sampling uniformly forward, but it isn't hard to use essentially the same structure to generate tokens conditioned on both preceding and following sequences. If you ran generation for tokens 1...n, and then ran m iterations of re-sampling internal token i based on (1..i-1, i+1..n), it would sometimes "fix" issues created initial generation pass. It would sometimes introduce new issues, which were fine upon original generation. Process-wise, it would look a lot like MCMC at generation-time.

The ability to "backtrack" does _not_ on its own add knowledge of reality, common sense, or "planning".

When a human edits, they're reconciling their knowledge of the world and their intended impact on their expected audience, neither of which the LLM has.

Re:Do large language models need sensory grounding for meani... by HervalFreire on Monday March 27, 2023 @02:30AM (#35323516)
>> LLMs have no knowledge of the underlying reality > They have no common sense & they can't plan their answer

If his arguments are entirely based on this, then it's not fully correct:

- GPT style language models try to build a model of the world: https://arxiv.org/abs/2210.13382

- GPT style language models end up internally implementing a mini "neural network training algorithm" (gradient descent fine-tuning for given examples): https://arxiv.org/abs/2212.10559

Re:Do large language models need sensory grounding for meani... by oezi on Monday March 27, 2023 @01:52AM (#35323317)
Sure they can. I even think GPT-3 does this by emitting a backspace token.
Re:Do large language models need sensory grounding for meani... by refulgentis on Monday March 27, 2023 @02:14AM (#35323442)
no
Re:Do large language models need sensory grounding for meani... by jackpirate on Monday March 27, 2023 @03:11AM (#35323747)
This is false. Standard sampling algorithms like beamsearch can "backtrack" and are widely used in generative language models.

It is true that the runtime of these algorithms is exponential in the length of the sequence, and so lots of heuristics are used to reduce this runtime in practice, and this limits the "backtracking" ability. But this limitation is purely for computational convenience's sake and not something inherent in the model.

Re:Do large language models need sensory grounding for meani... by taylorius on Thursday March 30, 2023 @04:38AM (#35369735)
I could be wrong, but I think the only probabilistic component of an LLM is the statistical word fragment selection at the end. Assuming this is true, one could theoretically run the program multiple times, making different fragment choices. This (while horribly inefficient) would allow a sort of backtracking.
Re:Do large language models need sensory grounding for meani... by oezi on Monday March 27, 2023 @01:52AM (#35323311)
I also don't get why this line of reasoning doesn't take into account that LLMs can just emit a backspace token in the same way that humans can ask for you to discard their statement.

The probability of a correct answer doesn't have to decrease in the length of the answer.

Epistemologically it certainly makes sense that you need to run a randomized controlled trial at some point to ascertain facts about nature, but alas we humans very rarely do. We primarily consume information and process that.

Re:Do large language models need sensory grounding for meani... by Hendrikto on Monday March 27, 2023 @06:03AM (#35324786)
These LLMs are trained with lots and lots of text from many sources, 99.9+% of which do not contain backspaces. They could theoretically output them, given the right tokenizer, but they don‘t.

And they especially don‘t do it to correct errors.

Re:Do large language models need sensory grounding for meani... by cma on Tuesday March 28, 2023 @06:30PM (#35348927)
They could potentially be trained on applying diffs though. Wikipedia has full source control version history of how things evolved, and so do most of the source code corpuses they use like github.

Google has keystroke by keystroke history of all google docs, could pull an Adobe Stock and opt everyone in with some TOS update but the blowback would be huge.

Re:Do large language models need sensory grounding for meani... by HervalFreire on Monday March 27, 2023 @01:52AM (#35323318)
That probability formula is too general. It can practically model everything. Yan is just hiding behind epsilon. It's like assigning a probability to the origin of the universe. You can just make shit up with a algebraic letter representing a probability.

To illustrate the absurdity of it consider this:

I have a theory for everything. I can predict future. The mathematical equation for that is simply (1 - EPSILON) where EPSILON is the probability of any event not happening. What about the probability of a event happening as a result of the first event? Well that's (1 - EPSILON)^2.

Since this model can model practically everything it just means everything diverges and everything fundamentally can't be controlled.

Really? No. He just tricked himself. The Key here is we need to understand WHAT EPSILON is. We don't, and Yan bringing this formula up ultimately says nothing about anything because it's too general.

Not to mention, Every token should have a different probability. You can't have the same epsilon for every token. You have zero knowledge of the probabilities of every single token therefore they cannot share the same variable, unless you KNOW for a fact the probabilities are the same.

It should be: P_n(Correct) = (1 - Epsilon_n)^n

Re:Do large language models need sensory grounding for meani... by HervalFreire on Monday March 27, 2023 @08:28AM (#35325802)
Error: formula above should be a recurrence relation:

P_n(correct) = (1 - Epsilon_n)*P_(n-1)(correct)

Where P_0 = (1 - Epsilon_0)

Re:Do large language models need sensory grounding for meani... by Al-Khwarizmi on Monday March 27, 2023 @04:29AM (#35324262)
I may be missing something as I don't claim to be as smart as LeCun, but to me that probabilistic argument is not even wrong.

The "probability e that a produced token takes us outside of the set of correct answers" is likely to vary so wildly due to a plethora of factors like filler words vs. keywords, hard vs. easy parts of the question, previous tokens generated (I know abuse of statistical independence assumptions is common and often tolerated, but here it's doing a lot of heavy lifting), parts of the answer that you can express in many ways vs. concrete parts that you must get exactly right, etc. that I don't think simplifying it as he does makes any sense.

Yes, I know, abstraction is useful, models are always simplifications, that probability doesn't need to be even close to a constant for the gist of the argument to stand. But everything can be bad in excess and in this case, the simplification is extreme. One can very easily imagine a long answer where the bulk of that probability is concentrated into a single word, while the rest have near-zero probability. Under such circumstances, I don't think his model is meaningful at all.

Connecting with your comment, if someone made that kind of claim about humans, I suppose most people would find it ridiculous (or at the very least, meaningless/irrelevant). With LLMs we find it more palatable, because we are primed to think about LLMs in terms of "probability of generating the next token". But I don't think it really makes much more sense for LLMs than for humans, as the problems with the argument are more in how language works than in how the words are generated.