News for hackers. Stuff that matters.
Re:Better and Faster Large Language Models via Multi-Token P... by throw310822 on Wednesday May 01, 2024 @05:26AM (#40221130)
Apologies in advance for the super naive question; but assuming that we can create vectors that encode for the meaning of entire sentences, what prevents us from training llms to predict those vectors instead of single words?
Re:Better and Faster Large Language Models via Multi-Token P... by faabian on Wednesday May 01, 2024 @06:00AM (#40221311)
Author here -- that's a very good point and as I understand work in progress in different teams. Training autoencoders for language is actually super easy given the small amount of information contained in text (compared to vision/video), the hard part is making the model focus on the semantic part if all signal we have comes from exact match in token space. Hence Yann LeCun's ideas on joint embedding predictive architectures. Note also that there is always a trade-off between auxiliary tasks giving more signal but shifting the focus. In our case, we noticed degradation if the number of predicted tokens is too high. So latent prediction methods need to sort out what is useful.
Re:Better and Faster Large Language Models via Multi-Token P... by mike_hearn on Wednesday May 01, 2024 @06:15AM (#40221413)
Aren't the models already doing this, in a way? We know they can do things like write rhyming poems and song lyrics that do make perfect sense, so at some point the activations must be encoding some sort of overall plan for the upcoming sentences, even if maybe every word isn't predicted yet.
Re:Better and Faster Large Language Models via Multi-Token P... by mjburgess on Wednesday May 01, 2024 @06:43AM (#40221585)
> so at some point the activations must be encoding some sort of overall plan for the upcoming sentences

This isn't obviously the case, compare this "intelligent designer" view with evolution: there was no prior plan for rabbits. it's sufficient to create the appearance of design that sequential steps are simply probabilistically modulated by prior ones.

Consider a continuation of "the cat..." merely a distribution over all possible words suffices to create the illusion of a plan, suppose: "the cat sat..." then, "on.., the..." etc. follow from the training data.

I think there's a strong argument against trying to model entire sentences exactly because the system isn't modelling semantics: one should expect accuracy to drop off a cliff if there is no actual plan. ie., predicting "sat on the mat" from "cat" shouldnt be a valid prediction, because of the infinite number of possible continuations that as a whole is terrible (eg., what about "chased the mouse" etc.). The space of all possible sentences to continue from "the cat" is infinite, which much of that space actually useful; whereas the number of words is very small, very fininte, and many of them not useful.

The only reason that "the cat sat..", "the cat sat on..." is reasonable is because each sequential word can be modulated by the prompt to seem as if planned.

Re:Better and Faster Large Language Models via Multi-Token P... by edmara on Wednesday May 01, 2024 @12:30PM (#40225543)
The modelling is advanced enough that you can't fundamentally distinguish it from (lossy, limited) planning in the way you're describing.

If the KQV doesn't encode information about likely future token sequences then a transformer empirically couldn't outperform Markov text generators.

Re:Better and Faster Large Language Models via Multi-Token P... by mjburgess on Wednesday May 01, 2024 @01:52PM (#40226740)
No one is spending $10-50mil building a markov text model of everything ever digitised; if they did so, their performance would approach a basic LLM.

Though, more simply, you can just take any LLM and rephrase it as a markov model. All algorithms which model conditional probability are equivalent; you can even unpack a NN as a kNN model or a decision tree.

They all model 'planning' in the same way: P(C|A, B) is a 'plan' for C following A, B. There is no model of P("A B C" | "A B"). Literally, at inference time, no computation whatsoever is performed to anticipate any future prediction -- this follows both trivially form the mathematical formalism (which no one seems to want to understand); or you can also see this empirically: inference time is constant regardless of prompt/continuation.

The reason 'the cat sat...' is completed by 'on the mat' is that it's maximal that P(on|the cat sat...), P(the|the cat sat on...), P(mat|the cat sat on the...)

Why its maximal is not in the model at all, nor in the data. It's in the data generating process, ie., us. It is we who arranged text by these frequencies and we did so because the phrase is a popular one for academic demonstrations (and so on).

As ever, people attribute "to the data" or worse, "to the LLM" no properties it has.. rather it replays the data to us and we suppose the LLM must have the property that generates this data originally. Nope.

Why did the tape recorder say, "the cat sat on the mat"? What, on the tape or in the recorder made "mat" the right word? Surely, the tape must have planned the word...

Re:Better and Faster Large Language Models via Multi-Token P... by edmara on Thursday May 02, 2024 @06:24AM (#40234568)
>Why it's maximal is not in the model at all, nor the data

>It replays the data to us and we suppose the LLM must have the property that generates this data originally.

So to clarify, what you're saying is that under the hood, an LLM is essentially just performing a search for similar strings in its training data and regurgitating the most commonly found one?

Because that is demonstrably not what's happening. If this were 2019 and we were talking about GPT-2 it would be more understandable but SoTA LLMs can in-context learn and translate entire languages which aren't in their dataset.

Also RE inference time, when you give transformers more compute for an individual token, they perform better https://openreview.net/forum?id=ph04CRkPdC

Re:Better and Faster Large Language Models via Multi-Token P... by faabian on Wednesday May 01, 2024 @06:30AM (#40221504)
Yes. Otherwise next-token models wouldn't be nearly as good as they are. But the question is how to train these capabilities most efficiently! We had some interesting findings on how with increasing model/dataset scale/data quality, capabilities can move from "only learnable with multi-token prediction" to "indifferent" and "multi-token prediction actually hurts". This depends on the capability itself, induction e.g. matures way earlier in this sense than code generation capabilities.
Re:Better and Faster Large Language Models via Multi-Token P... by mike_hearn on Wednesday May 01, 2024 @07:59AM (#40222046)
Is it possible that anti-scaling effect occurs because you are removing some middle layers to free up space for the extra output heads? I only scanned the paper quickly but what happens if you treat the technique as strictly additive and don't keep parameter sizes fixed?
Re:Better and Faster Large Language Models via Multi-Token P... by flawsofar on Wednesday May 01, 2024 @09:29AM (#40222935)
In case you’re thinking that rhyming requires planning, that’s just as silly as a rabbit tanning.

You can make things up as you go, and the constraints emerge from the flow.

Re:Better and Faster Large Language Models via Multi-Token P... by gbasin on Wednesday May 01, 2024 @09:44AM (#40223122)
great comment
Re:Better and Faster Large Language Models via Multi-Token P... by marcyb5st on Wednesday May 01, 2024 @05:44AM (#40221232)
Not a stupid question in my opinion.

The problem is that once you have the vectors representing the answers you need something like another model that goes back to a word representation of said answers. Something like a diffusion model but for text. Additionally, the function that this diffusion model will approximate won't be injective, but at best surjective and at worst not even a function (in the mathematical meaning) since many textual represantions are possible given an embedding, and most of those won't be valid (not grammatically valid, no sense sentences, ...).

Finally remember that the embeddings are a "lossy" representation of some datum and so the inverse function will lose a lot of the nuances/context/... .

LLMs avoid the problems above by predicting the next (now next n tokens) in a way that is self consistent with the query and the previous n tokens, so the function they approximate should be mostly surjective.

Re:Better and Faster Large Language Models via Multi-Token P... by throw310822 on Wednesday May 01, 2024 @06:17AM (#40221428)
> The problem is that once you have the vectors representing the answers you need something like another model that goes back to a word representation of said answers. Something like a diffusion model but for text.

Could it be just a smaller llm that takes as input both the semantic vector and the prompt, and is trained to predict the output tokens based on those? A model with high linguistic abilities and very little reasoning skills.

Re:Better and Faster Large Language Models via Multi-Token P... by marcyb5st on Wednesday May 01, 2024 @06:38AM (#40221557)
I think what you suggest would be very similar to a encoder-decoder architecture, which has been abandoned in favor of decoder-only architectures (https://cameronrwolfe.substack.com/p/decoder-only-transforme...). So I am guessing that what you suggest has already been tried and didn't work out, ut not sure why (the problems I mentioned above or something else).

Sorry, that's where the limit of my knowledge is. I work on ML stuff, but mostly on "traditional" deep learning and so I am not up to speed with the genAI field (also, the sheer amount of papers coming out makes it basically impossible stay up to date of you're not in the field).

Re:Better and Faster Large Language Models via Multi-Token P... by HanClinto on Wednesday May 01, 2024 @09:54AM (#40223285)
> Something like a diffusion model but for text.

This actually sounds amazingly useful.

Re:Better and Faster Large Language Models via Multi-Token P... by soulofmischief on Wednesday May 01, 2024 @12:56PM (#40225957)
I've played around with the concept in the past but there are some issues yet to be solved to make them more practical than current generation LLMs.

People are working on it though: https://arxiv.org/pdf/2305.09515

Re:Better and Faster Large Language Models via Multi-Token P... by wangii on Wednesday May 01, 2024 @07:45AM (#40221948)
the problem is then the total number of computation drops dramatically therefore leads to much less "thinking" power. i think the idea originated from an understanding that when we write/speak, we have an overall idea. my current hypothesis is it's probably an illusion.

you may want to search for "filler" papers to read.

Re:Better and Faster Large Language Models via Multi-Token P... by jerpint on Wednesday May 01, 2024 @06:08AM (#40221368)
My understanding is that tokenization is part of the bottleneck. When you break up a sentence to tokens, each token gets a vector representation. The dictionary of all tokens would be infinite if it was at the sentence level
Re:Better and Faster Large Language Models via Multi-Token P... by faabian on Wednesday May 01, 2024 @06:24AM (#40221470)
Vectors can do what one-hot vectors cannot do -- no one said inputs need to be rows from an token_id -> vector embeddings map. Basically, we are doing this already by moving from one-hot vectors to n-tuples of one-hot vectors, increasing the effective vocabulary size from V to V^n.
Re:Better and Faster Large Language Models via Multi-Token P... by bjourne on Wednesday May 01, 2024 @06:47AM (#40221612)
That's hierarchical prediction. On one level you predict the style of the paragraph, on another the tone and form of the sentence, and on the third the next word. However, this form of prediction is quite difficult since predictions from the layers affect each other.
Re:Better and Faster Large Language Models via Multi-Token P... by yorwba on Wednesday May 01, 2024 @05:31AM (#40221162)
You still need to convert between words and sentence vectors somehow. You could try using a faster model for that, but I suspect that the output quality will suffer.
Re:Better and Faster Large Language Models via Multi-Token P... by magicalhippo on Wednesday May 01, 2024 @06:42AM (#40221577)
LLMs somehow need to do this anyway, implicitly or not. Has anyone tried to do it more explicitly?

That is to break out the idea that characters are formed into words, and words into sentences, and a sentence is a sequence of "concepts" for the lack of a better description.

So have one NN which takes a sequence of tokens and predicts an moderately-dimensional "word vector", which is fed into another which predicts a high-dimensional "concept vector".

Then the "thinking layer" would map a sequence of "concept vectors" to "concept vectors", and then you'd have some layers which does the reverse of the input layers to output tokens which can be printed.

Thought being that by splitting it up like this you could swap out the decode and encode layers independently to translate, for example, and so on.

Just a shower thought.

Re:Better and Faster Large Language Models via Multi-Token P... by flawsofar on Wednesday May 01, 2024 @09:40AM (#40223079)
A sentence is both a sequence and a tree of phrases. Phrases have a head and one or more valents; they’re relations.

If you wanted to create an embedding algorithm for phrases, you could and you could throw a transformer at it.

I don’t know how you get the output of higher levels to diffuse to phrase and word levels.

Re:Better and Faster Large Language Models via Multi-Token P... by everforward on Wednesday May 01, 2024 @10:15AM (#40223553)
Also a noob here, if we encoded, trained on and synthesized sentence vectors, wouldn’t that move the AIs ability to create novel things up from sentences to words?

I.e. we currently operate on words (roughly) so the AI can only use words it knows but can synthesize unique sentences from words. If the AI operates on sentences, wouldn’t it only be able to regurgitate sentences it has seen before? So it could synthesize novel paragraphs, but not sentences?

I’m not convinced that sentences are a useful abstraction for AI (in English, anyways). They’re barely useful to humans. Check out your average chat conversation, email, YouTube comment, etc. There’s a very good chance the sentences aren’t actually sentences, or that they haven’t even bothered to use punctuation.

I just don’t think sentences map to a semantic device. A sentence could be two words or half an English paper depending on the writer. It could traverse a half dozen ideas or a single one. Where a sentence ends generally is more about the writer than the semantics.