News for hackers. Stuff that matters.
Re:Gemma 2: Improving Open Language Models at a Practical Si... by jakobov on Thursday June 27, 2024 @12:47PM (#40812470)
Gotcha. That makes sense. Thanks!

What are the theories as to why this works better than training on a larger quantity of non-simulated tokens?

Is it because the gradient from the non-simulated tokens is too noisy for a small model to model correctly?

There are no comments on this comment yet.