Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on Gemma 2: Improving Open Language Models at a Practical Size [pdf]. Parent comment
What are the theories as to why this works better than training on a larger quantity of non-simulated tokens?
Is it because the gradient from the non-simulated tokens is too noisy for a small model to model correctly?
There are no comments on this comment yet.
What are the theories as to why this works better than training on a larger quantity of non-simulated tokens?
Is it because the gradient from the non-simulated tokens is too noisy for a small model to model correctly?