News for hackers. Stuff that matters.
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String by cpldcpu on Tuesday July 09, 2024 @02:07AM (#40912948)
Searching for the string on google yields hundreds of hits. Likely that it appears in recent webscrape-data.
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String by andrewflnr on Tuesday July 09, 2024 @04:01AM (#40913605)
The model knows the original context too, though. So it doesn't make a lot of practical difference if it found it through a secondary source.
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String by falcor84 on Thursday July 11, 2024 @05:04PM (#40940597)
But the discussion has mostly centered around whether Anthropic have properly scrubbed benchmark data out of their training corpus before training. So it seems to me that even if they did it properly (maybe especially so), the presence of the canary string in Claude's output is likely if it saw it in a secondary source (e.g. a blog post), which should then be fair game, no?
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String by tracerbulletx on Thursday July 11, 2024 @06:05PM (#40941018)
It makes literally all the difference that matters?