Re:Chinchilla Scaling: A Replication Attempt
by saurabh20n on Thursday April 18, 2024 @09:33PM
(#40082602)
Looks like you’re one of the authors.
It would be nice if you could post if the actual data matches your reconstruction—now that you have it in hand. Would help us not worry about the data provenance and focus on the result you found.
Re:Chinchilla Scaling: A Replication Attempt
by ege_erdil on Friday April 19, 2024 @02:27AM
(#40083956)
we're not sure if the actual data exactly matches our reconstruction, but one of the authors pointed out to us that we can exactly reproduce their scaling law if we make the mistake they made when fitting it to the data
what they did was to take the mean of the loss values across datapoints instead of summing them and used L-BFGS-B with the default tolerance settings, so the optimizer terminated early, and we can reproduce their results with this same mistake
so our reconstruction appears to be good enough
also, we didn't reconstruct the data using a ruler, you can automate that entire process so that it's much more reliable than that