Re:Show HN: Beating GPT-4 on HumanEval with a fine-tuned Cod...
by dragonwriter on Friday August 25, 2023 @08:16PM
(#37268630)
> when a model contains a good enough distilled representation of arguably all the code out there, does it really matter whether it can generalise OOD?
If its contaminated by the test set being in the model’s training set, then the test is no longer (assuming it was in the first place) a valid measure of whether the model has “a good enough distilled representation of arguably all the code out there”.
If its contaminated by the test set being in the model’s training set, then the test is no longer (assuming it was in the first place) a valid measure of whether the model has “a good enough distilled representation of arguably all the code out there”.