Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String
by andrewflnr on Tuesday July 09, 2024 @04:01AM
(#40913605)
The model knows the original context too, though. So it doesn't make a lot of practical difference if it found it through a secondary source.
[ Parent
| on HN ]
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String
by falcor84 on Thursday July 11, 2024 @05:04PM
(#40940597)
But the discussion has mostly centered around whether Anthropic have properly scrubbed benchmark data out of their training corpus before training. So it seems to me that even if they did it properly (maybe especially so), the presence of the canary string in Claude's output is likely if it saw it in a secondary source (e.g. a blog post), which should then be fair game, no?
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String
by tracerbulletx on Thursday July 11, 2024 @06:05PM
(#40941018)
It makes literally all the difference that matters?