Gemma 2: Improving Open Language Models at a Practical Size [pdf]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by WhitneyLand on Thursday June 27, 2024 @01:51PM (#40813130)

The paper suggests on one hand Gemma is on the same Pareto curve as Llama3, while on the other hand seems to suggest it’s exceeded its efficiency.

Is this a contradiction or am I misunderstanding something?

Btw overall very impressive work great job.

[ Parent | on HN ]

Re:Gemma 2: Improving Open Language Models at a Practical Si... by alekandreev on Thursday June 27, 2024 @02:39PM (#40813599)

I think it makes sense to compare models trained with the same recipe on token count - usually more tokens will give you a better model.

However, I wouldn't draw conclusions about different model families, like Llama and Gemma, based on their token count alone. There are many other variables at play - the quality of those tokens, number of epochs, model architecture, hyperparameters, distillation, etc. that will have an influence on training efficiency.

[ Parent | on HN ]