Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String
by jhugo on Friday July 12, 2024 @04:27AM
(#40943663)
So just to understand this correctly, you're suggesting that they have some people/processes dedicated to keeping track of any canary string published by anyone (or some defined subset of "anyone"), and updating their ingest to ignore any documents that contain those strings?
[ Parent
| on HN ]
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String
by PostOnce on Friday July 12, 2024 @07:58AM
(#40944716)
I explained above that they don't have to care, but if they're going to want to be included on major industry benchmarks like those created by Google, they should probably go to the effort of ignoring the half dozen or so notable benchmarks with canary GUIDs. Blacklisting a GUID is not difficult, not even at web-scale.