Re:Wavecoder – a CodeLLM with 6.7B params scoring just behin...
by eightysixfour on Thursday January 18, 2024 @11:18AM
(#39043620)
That’s to prevent overfitting on their dataset, it is not to prevent overfitting on the test data, which is likely in their dataset.
You basically cannot beat GPT-4 on broad reasoning tasks, which the tests are designed to cover, without having some of the tests leaking into training dataset. There simply aren’t enough parameters and isn’t enough training to make that possible.
I'm way out of my league here so I have no opinion on whether or not that actually addresses overfitting.
(that quote probably doesn't capture their intention - just a pointer into the paper)