News for hackers. Stuff that matters.

A comment on The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT. Parent comment

Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by novok on Sunday April 09, 2023 @10:23PM (#35508538)
Well you are comparing 165B and 1000B parameter models to 30B and 60B parameter models. That is about 3 to 33x larger, not an apples to apples comparison.
[ Parent | on HN ]
Re:The LLama Effect: Leak Sparked a Series of Open Source Al... by crakenzak on Sunday April 09, 2023 @10:47PM (#35508682)
I agree with your general statement, but if one thing the recent DeepMind Chinchilla research paper showed us is that the size of the model (number of parameters) is much less of the determinant of model performance & quality than the amount of high-quality data (number of high-quality tokens).

Their 70B chinchilla model significantly outperforms the 175B GPT3 model.

Possibly where OpenAI has a leg up is their high-quality data sourcing & curating infrastructure and their RLHF mechanisms.

Paper: https://arxiv.org/abs/2203.15556

[ Parent | on HN ]