News for hackers. Stuff that matters.

A comment on Pretraining on the Test Set Is All You Need. Parent comment

Re:Pretraining on the Test Set Is All You Need by tveita on Saturday October 28, 2023 @02:40PM (#38052291)
How do you even put together an open evaluation benchmark without it being slurped up into the training set the minute someone puts it on Github?

Maybe it needs to be obfuscated, or all the questions should be obscene so they don't pass the content filtering.

[ Parent | on HN ]
Re:Pretraining on the Test Set Is All You Need by gwern on Saturday October 28, 2023 @03:37PM (#38052723)
There are 'canary tokens' that you can add to your dataset which are supposed to be respected by self-regulating scrapers & responsible ML practitioners to grep them out before doing any training. How well it works is unknown, of course...
[ Parent | on HN ]