Re:Can LLMs learn from a single example?
by armatav on Thursday September 07, 2023 @07:57PM
(#37427376)
That's super interesting, I wonder if dropout applied in a certain pattern of segregation could induce more neuron dependence, instead of less. Or do some anti-dropout where a dedicated portion of the model is used. Not sure why you'd want that, but it might be interesting to explore.
From a physics perspective it’s entropy: There are just more local minima that have neurons code multiple things.
I suspect that dropout and similar tricks also play a role in this: Removing connections during training means that pathways need to be redundant somewhat.