News for hackers. Stuff that matters.

A comment on Chinchilla’s Death. Parent comment

Re:Chinchilla’s Death by visarga on Tuesday September 05, 2023 @01:33AM (#37388213)
> there are no regularizers in any modern LLMs.

Using a large & diverse training set is the best regulariser, but I think there is also weight decay and dropout in transformers

[ Parent | on HN ]
Re:Chinchilla’s Death by euclaise on Tuesday September 05, 2023 @10:27PM (#37400712)
RWKV also uses some sort of L2-esque regularization, which was supposedly an idea taken from PaLM (although I can't find a source on this point, other than some message in the RWKV discord)
[ Parent | on HN ]