News for hackers. Stuff that matters.
Re:What We Know About LLMs (A Primer) by moffkalast on Tuesday July 25, 2023 @07:08PM (#36870983)
Isn't dropout just there to avoid overfitting? This is more like a mixture of experts type architecture.
Re:What We Know About LLMs (A Primer) by esafak on Tuesday July 25, 2023 @08:22PM (#36871542)
That is one lens to view it through. Co-adaptation reduction is another, and it is an intuitive one: generalization ability is improved if a neuron has to be predictive in multiple contexts, and can't rely on other neurons to lift all the weight, if you pardon the pun.

Improving neural networks by preventing co-adaptation of feature detectors https://arxiv.org/abs/1207.0580