News for hackers. Stuff that matters.
Re:Can LLMs learn from a single example? by armatav on Wednesday September 06, 2023 @06:28PM (#37412274)
Is that necessarily the case for massive LLMs, or could there be a more refined grouping?
Re:Can LLMs learn from a single example? by niemandhier on Thursday September 07, 2023 @05:08AM (#37416447)
I am not sure if the origin of polysyntacicity is fully understood.

From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things.

I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.

Re:Can LLMs learn from a single example? by armatav on Thursday September 07, 2023 @07:57PM (#37427376)
That's super interesting, I wonder if dropout applied in a certain pattern of segregation could induce more neuron dependence, instead of less. Or do some anti-dropout where a dedicated portion of the model is used. Not sure why you'd want that, but it might be interesting to explore.