News for hackers. Stuff that matters.
Re:Can LLMs learn from a single example? by jph00 on Tuesday September 05, 2023 @08:56PM (#37400003)
Thank you for posting this to HN! :D

I'm one of the authors of this post -- Johno & I found it really interesting looking into this curious issue of rapid memorization from LLMs. I've been working with neural nets for 30 years, and fine-tuning language models since 2017, and this behavior is most surprising to me! Other folks have seen it in LLMs too, although I haven't seen a analysis of this kind before (although we might have missed something).

Let me know if you have any questions or thoughts.

Re:Can LLMs learn from a single example? by og_kalu on Tuesday September 05, 2023 @09:09PM (#37400120)
In the palm-e paper (https://palm-e.github.io/), when they try to unfreeze and train the LLM on new image data only, there is expectedly a lot of CF on NLP tasks but very interestingly, the effect diminishes greatly with the scale of the LLM prior to training.

From an average -87.3% performance drop on the 12B model to -61.6% on the 84B model then just -3.9% on the 562B model. Felt like we were just shy of an insight breakthrough here.

Is avoiding CF potentially just a matter of sheer scale ?

Re:Can LLMs learn from a single example? by jph00 on Tuesday September 05, 2023 @09:11PM (#37400137)
I think our experiments actually don't show catastrophic forgetting! The accuracy does not decrease as loss gets worse -- it's simply getting over-confident.

So I'm not even sure we're showing any problem to solve here -- it might be more of a opportunity, in fact!

Re:Can LLMs learn from a single example? by Yenrabbit on Tuesday September 05, 2023 @09:14PM (#37400166)
It does start getting worse at some point right?
Re:Can LLMs learn from a single example? by jph00 on Tuesday September 05, 2023 @09:19PM (#37400214)
I'm sure eventually it would, but we haven't gotten to that point yet in our training.
Re:Can LLMs learn from a single example? by minihat on Wednesday September 06, 2023 @12:18AM (#37401301)
Cross-entropy loss can start getting worse due to the model becoming less calibrated, even as the classification accuracy continues to improve. I first heard that here: https://arxiv.org/abs/1706.04599

Is this 'overconfidence' the leading explanation as to why LLMs continue to show qualitative improvement even after their test loss levels off?

Re:Can LLMs learn from a single example? by alekseiprokopev on Wednesday September 06, 2023 @06:30AM (#37403320)
Is it possible to somehow modify the sampling from the model to account for that?
Re:Can LLMs learn from a single example? by samstave on Wednesday September 06, 2023 @12:37AM (#37401389)
Plz eli5 catastrophic forgetting,

I assume this means losing all the energy and compute input for a model to know, perform, infer on inputs already indexed(?) (What is the proper term here?)

But is this the premise -you lose all prior investment of resource to a (I don't know the term for an AI archetype of knowledge) {btw, I love the embedded entomology of knowledge

"The ledger of things that we KNOW"}

Re:Can LLMs learn from a single example? by tarvaina on Wednesday September 06, 2023 @12:59AM (#37401482)
Suppose we have trained a model to perform a certain set of tasks. Later we would want to teach it a new task. Catastrophic forgetting means that teaching it a new task makes it unlearn some or all of its earlier tasks.

It occurs because training changes the weights of the model. The earlier set of weights was good for the previous tasks. The new set of weights is only good for the new task. Usually special care must be taken to overcome catastrophic forgetting.

Re:Can LLMs learn from a single example? by samstave on Wednesday September 06, 2023 @01:03AM (#37401501)
Can it be taught "contextual matrices" where by it builds a new layer of construct but preserves the other, then cross learns between parameters or something (sorry for my poor lexicon, I'm wet-learning :-)

But imagine all LLMs in a macro view like a sponge entity

Re:Can LLMs learn from a single example? by tinco on Wednesday September 06, 2023 @03:31AM (#37402267)
We wouldn't know how to construct those matrices because we don't know where in the layers what knowledge is represented. One thing that helps a little bit is freezing the lower layers, so at least the model won't forget its most fundamental knowledge.

Note that the only reason that things are catastrophically forgotten, is that the original examples are not shown again. If the model learns in a single shot, there might simply be no time to show both the old and the new examples. I don't think it would have a significant effect or else we'd know about this effect a lot sooner (i.e. the training of these LLM's would get less effective from a certain point)

Re:Can LLMs learn from a single example? by jacquesm on Wednesday September 06, 2023 @05:04AM (#37402810)
You could simulate this by selectively locking and unlocking 'banks' of weights from a larger model to keep the influence there during training and to avoid losing them. Sort of a selective write-protect.
[dead]
Re:Can LLMs learn from a single example? by tinco on Thursday September 07, 2023 @11:41AM (#37420613)
That's not how LLM's work. LLM's complete documents, they don't make statements about LLM's unless you explain to them how they should do it and give them all the information they need. If you could extract the information from an LLM well enough to supply that to an LLM with an explanation on how to summarize the behaviour of the LLM to a human, we would have already done that to a PhD student instead. A PhD student is a little bit slower than an LLM, but they require a lot less explanation.

In any case, looking at and understanding how a neural network encodes information is like gene editing. Perhaps you could isolate a gene in the human genome that achieves something interesting like giving a child blue eyes. But even if you would do that, there's a chance you break something else if you modify that gene and give the child health risk. Since all neurons in a deep neural network are interconnected, there is a butterfly effect in it that makes them inherently somewhat of a black box.

Re:Can LLMs learn from a single example? by antupis on Wednesday September 06, 2023 @04:06AM (#37402468)
I think some cases CF would be even good eg you want llm that produces only valid json data as output.
Re:Can LLMs learn from a single example? by josephg on Wednesday September 06, 2023 @05:47AM (#37403060)
Yeah, this is essentially how finetuned models work. If you fine tune stablediffusion to produce anime images, it might forget how to produce images in any other style. But it will become much better at anime images than the base model. If anime images are the art style you’re after, this is a good trade. Same with fine tuning LLMs for SQL or whatever.
Re:Can LLMs learn from a single example? by 3abiton on Wednesday September 06, 2023 @01:03AM (#37401500)
Awesome investigative work, what's the opportunity though, I don't get it
Re:Can LLMs learn from a single example? by jph00 on Wednesday September 06, 2023 @05:14AM (#37402871)
We don't know. It's a report of some early experimental results. Our hope is that it will stimulate discussion and further research and development.
Re:Can LLMs learn from a single example? by mirekrusin on Wednesday September 06, 2023 @06:41AM (#37403375)
It looks like something clicks in place.
Re:Can LLMs learn from a single example? by vvrm on Wednesday September 06, 2023 @02:04AM (#37401778)
I have been training a natural intelligence model for 3 years now and she still doesn’t get nuance. Things are either good or bad in her book: nothing in between. My plan is to let her train with binary good/bad labels till the age of 5 and then start smoothing the labels after that. Wonder if that works for your AI.
Re:Can LLMs learn from a single example? by vineyardmike on Wednesday September 06, 2023 @02:53AM (#37402030)
This took a couple reads, but it’s funny. The bad news is that I’ve been training mine for 17 years and nuance is still something that needs more training.
Re:Can LLMs learn from a single example? by tudorw on Wednesday September 06, 2023 @04:34AM (#37402655)
in my mind I've built an 'emotional engine' to add nuance to models understanding, take something like Plutchik's wheel of emotions and create a high quality multi-modal dataset based on that structure, given our current technology takes inspiration from the brain, it would seem like having discrete models specialising in particular aspects of 'intelligence' that are then organised into a mixture of experts is an interesting area to explore, and perhaps more accessible as smaller models require less resources.
Re:Can LLMs learn from a single example? by kordlessagain on Wednesday September 06, 2023 @11:43AM (#37406775)
I have code stubbed out for this in mitta.us. It has 9 states, based on the Plutchik wheel, with emojis for the states. States drive temp and a few other things and drop the state into prompts.
Re:Can LLMs learn from a single example? by tudorw on Thursday September 07, 2023 @03:49AM (#37415915)
Interesting, do you have a mailing list or way I can be notified of progress?
Re:Can LLMs learn from a single example? by kordlessagain on Saturday September 09, 2023 @01:11PM (#37447632)
You can signup: https://mitta.us/

The accounts aren't wired up by default to the AI and I am refactoring the templating system right now, but you can definitely start storing and searching things.

Re:Can LLMs learn from a single example? by joquarky on Thursday September 07, 2023 @01:12AM (#37414986)
Re:Can LLMs learn from a single example? by tudorw on Thursday September 07, 2023 @03:48AM (#37415911)
very interesting, thanks.
Re:Can LLMs learn from a single example? by TeMPOraL on Wednesday September 06, 2023 @07:50AM (#37403759)
Related trick: I found that training two Natural Intelligence (NI) models in parallel, and having them train each other for most of the time, leads to significant leaps in capabilities. Notably, when one NI picks up a skill, it often results in spontaneous transfer learning - the other NI picks that skill up very quickly, much faster than it would through direct training.

This scales well, too. There are facilities that provide services of co-hosting and cross-training up to ~two dozen NI models in a shared environment - in my experience, this provides similar training benefits to running multiple NIs on your own, at fraction of the cost.

(The facilities are exploiting some neat economies of scale. Talking to some employees, I learned that the transfer learning and co-activation are embarrassingly scalable: if you get two-three NIs to pick up a thing, all the rest immediately follow.)

Re:Can LLMs learn from a single example? by Yenrabbit on Tuesday September 05, 2023 @09:15PM (#37400177)
Ooh interesting, thanks for sharing!
Re:Can LLMs learn from a single example? by theonlybutlet on Wednesday September 06, 2023 @07:04AM (#37403483)
What does CF stand for?
Re:Can LLMs learn from a single example? by d4rkp4ttern on Wednesday September 06, 2023 @07:38AM (#37403672)
Catastrophic Forgetting
Re:Can LLMs learn from a single example? by theonlybutlet on Wednesday September 06, 2023 @08:53AM (#37404388)
Thank you
Re:Can LLMs learn from a single example? by t-vi on Thursday September 07, 2023 @01:57PM (#37422924)
> Is avoiding CF potentially just a matter of sheer scale ?

My intuition would be that you get more orthogonal directions to the gradient (of previous samples) if you have larger model.

Re:Can LLMs learn from a single example? by azg123 on Tuesday September 05, 2023 @09:30PM (#37400311)
Super interesting! Another area that I've seen these types of loss curves are recommendation models: https://arxiv.org/pdf/2209.06053.pdf
Re:Can LLMs learn from a single example? by ilaksh on Tuesday September 05, 2023 @09:48PM (#37400461)
What is the base model? I think that was a big oversight to leave that out and attribute this to LLMs in general.

Although I am not a researcher, it is obvious to me that not all LLMs are the same architecture, and I think that even ones with similar architecture can evolve to functionally operate quite differently on the same inputs.

Yet most articles seem to refer to LLMs as if they were just one architecture and model.

Re:Can LLMs learn from a single example? by n9Mtq4 on Tuesday September 05, 2023 @09:52PM (#37400489)
Very cool. This came up in a huggingface transformers issue a while ago and we also determined memorization to be the likely reason. It's nice to see someone else reach the same conclusion.

https://github.com/huggingface/transformers/issues/18730

Re:Can LLMs learn from a single example? by jwuphysics on Tuesday September 05, 2023 @10:01PM (#37400548)
Hi Jeremy, always a fan of your work! Just a technical note since it falls under my domain of expertise (astronomy) -- the example about MOND described here should actually have choice (E) as the correct answer!
Re:Can LLMs learn from a single example? by jwuphysics on Tuesday September 05, 2023 @10:11PM (#37400625)
In terms of the actual article -- really nice finding. Or I guess, nice set of experiments to decipher what lots of LLM researchers have been finding!

I've noticed somewhat similar behavior while training graph neural networks to model physical systems, except that it takes way longer than a single epoch to get there. Or course, there's no pretending involved with my GNNs, but the models do have very constrained representations, so once they start to figure out how to represent the physics at hand, the loss plummets dramatically.

Re:Can LLMs learn from a single example? by jph00 on Tuesday September 05, 2023 @10:58PM (#37400862)
As it happens I dug into this question in some detail a couple of weeks ago when analysing the dataset, including carefully reading the wikipedia page which the question comes from. AFAICT both D and E are kinda correct, but E isn't quite right because MOND doesn't entirely "eliminate the observed missing baryonic mass", but rather just reduces it from a factor of 10 to 2.

Is that not correct? (Of course I fully accept your expertise in this matter and this is just my curiosity, not trying to tell you you're wrong!)

Re:Can LLMs learn from a single example? by jwuphysics on Wednesday September 06, 2023 @09:23AM (#37404752)
Fascinating! I dug into the Wikipedia article, which cites a Scholarpedia article; the LLM answer seems to originate from a reference to this sentence [1]:

> So, MOND reduces the discrepancy in clusters at these radii to only a factor of ∼2−3 (Sanders, 1999; Ettori, et al., 2019)

So I think you're right, and today I learned something! I also checked if Stacy McGaugh had weighed in on this particular subject, and it seemed like there is still an issue for clusters [2], although interestingly the issue isn't mentioned in his latest blog post that summarizes the strengths/weaknesses with MOND [3]. Anyway, thanks for humoring me for a bit.

[1] http://www.scholarpedia.org/article/The_MOND_paradigm_of_mod... [2] https://tritonstation.com/2021/02/05/the-fat-one-a-test-of-s... [3] https://tritonstation.com/2023/06/27/checking-in-on-troubles...

Re:Can LLMs learn from a single example? by mannykannot on Wednesday September 06, 2023 @10:11AM (#37405426)
I believe neither MOND nor Condensed Dark Matter are theories exactly, so much as they are schemata for classes of theories. Both are struggling to produce a verified theory that accounts for all observations, and while the latter is much more widely regarded as likely being correct, MOND has not been conclusively falsified to everyone's satisfaction. I would guess that there are, at least in principle, MOND theories which work for galaxy clusters but have residual discrepancies when applied to galaxies.

If this is so, then a multi-choice question which conflates one particular MOND theory for MOND itself, and which depends on the specifics of that particular theory for selecting the 'correct' answer, is problematic: for one thing, it may make selecting the 'correct' answer more difficult for a student who has specific knowledge about the topic. This is just one of several problems with multi-choice questions, though, fortunately, it does not seem to have any bearing on the very interesting phenomenon you have discovered.

Re:Can LLMs learn from a single example? by ScoutOrgo on Tuesday September 05, 2023 @10:01PM (#37400549)
Hey Jeremy, it seems like you could calculate exactly how much a model learns in a single step by calculating the loss for a batch a second time (with no_grad) after the loss is calculated the first time and gradients are updated. This seems like it could produce interesting outputs when graphing the difference of first and second losses at the batch or observation/question level.
Re:Can LLMs learn from a single example? by startupsfail on Tuesday September 05, 2023 @11:14PM (#37400953)
Interesting, but you should show the example as concrete evidence, rather than hand waving arguments based on loss curves “evidence”.
Re:Can LLMs learn from a single example? by armatav on Wednesday September 06, 2023 @02:25AM (#37401886)
I wonder if you could perform inference, highlight the weights that were most used during that inference, grab the hottest 20%, freeze the rest of the model, and perform backpropagation solely on those to allow for more of this sort of rapid memorization behavior closer to the end user.

Like online learning in a way. But you do it during inference time.

There’s no way the entire model actually needs to be touched for something like “sky color is:” and “blue”.

Re:Can LLMs learn from a single example? by armatav on Wednesday September 06, 2023 @02:27AM (#37401888)
In fact I bet you could update like one or two neurons for certain concepts, and then transplant those neurons to another LLM to give it some idea of it. Like a literal brain transplant but for concepts.
Re:Can LLMs learn from a single example? by armatav on Wednesday September 06, 2023 @02:28AM (#37401897)
And you could identify these neurons using dropout techniques and repetitively querying the model against them.

Drop a set of neurons and there’s no change? Probably doesn’t contain the “sky color” concept.

Drop a set of neurons and the model freaks out, definitely conceptual neurons.

Rinse and repeat to find the distilled pattern across all the neurons.

You could train an LLM against the neuron graph to do this for you.

Re:Can LLMs learn from a single example? by niemandhier on Wednesday September 06, 2023 @02:33AM (#37401922)
Many neurons are polysynthactic, that makes interventions like the proposed difficult.
Re:Can LLMs learn from a single example? by armatav on Wednesday September 06, 2023 @06:28PM (#37412274)
Is that necessarily the case for massive LLMs, or could there be a more refined grouping?
Re:Can LLMs learn from a single example? by niemandhier on Thursday September 07, 2023 @05:08AM (#37416447)
I am not sure if the origin of polysyntacicity is fully understood.

From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things.

I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.

Re:Can LLMs learn from a single example? by armatav on Thursday September 07, 2023 @07:57PM (#37427376)
That's super interesting, I wonder if dropout applied in a certain pattern of segregation could induce more neuron dependence, instead of less. Or do some anti-dropout where a dedicated portion of the model is used. Not sure why you'd want that, but it might be interesting to explore.
Re:Can LLMs learn from a single example? by Angostura on Wednesday September 06, 2023 @07:41AM (#37403698)
As a lay-person, can I just say I appreciated the accessible writing style. Thanks!