News for hackers. Stuff that matters.

A comment on NanoGPT. Parent comment

Re:NanoGPT by belter on Wednesday January 11, 2023 @05:49AM (#34337087)
Model size does not necessarily correlates to quality of results.

"Chinchilla (70B) Greatly Outperforms GPT-3 (175B) and Gopher (280B)" - https://towardsdatascience.com/a-new-ai-trend-chinchilla-70b...

[ Parent | on HN ]
Re:NanoGPT by espadrine on Wednesday January 11, 2023 @06:14AM (#34337236)
An interesting outcome of the nanoGPT repo is this struggle to exactly match the Chinchilla findings[0], even after discussing it with the authors.

A larger discussion is that the scaling laws achieve loss-optimal compute time, but the pre-training loss only improves predictions on the corpus, which contains texts written by people that were wrong or whose prose was lacking. In a real system, what you want to optimize for is accuracy, composability, inventiveness.

[0]: https://github.com/karpathy/nanoGPT/blob/master/scaling_laws...

[ Parent | on HN ]
Re:NanoGPT by Der_Einzige on Wednesday January 11, 2023 @05:56AM (#34337127)
I highly doubt this in practice on a large scale. Outside of the common phenomena of "most large NNs are under trained" and "less better data is sometimes better than more worse data", there are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one.

I claim instead that we are still hardly scratching the surface with how we evaluate NLP systems. Also, some fields have straight up trash evaluation schemes. Summarization and ROGUE scores are totally BS and I find the claim that they even correlate with high quality summaries suspect. I say this with publications in the that subfield, so I have personal experience with just how crummy many summarizes are.

[ Parent | on HN ]
Re:NanoGPT by WithinReason on Wednesday January 11, 2023 @08:08AM (#34337953)
there are no other obvious mechanisms to explain why a smaller model with same or similar architecture would be better than a larger one.

Overfitting?

[ Parent | on HN ]
Re:NanoGPT by Der_Einzige on Wednesday January 11, 2023 @08:14AM (#34338012)
The consensus seems to be that the majority of LMs are undertrained not overfitting though.
[ Parent | on HN ]