News for hackers. Stuff that matters.
Re:Claude 3.5 Sonnet Reproduces BIG-Bench Canary String by PostOnce on Monday July 08, 2024 @09:45PM (#40911771)
No, not "untraining" after the fact, but scrubbing the benchmark data from the dataset prior to training. That's absolutely trivial.

Take the set of benchmarks you care about, and then as you build a training dataset (by scraping, or whatever), you scrub each new item for benchmark questions (or just discard that entire item/webpage/whatever for being contaminated).

Otherwise you're negligently and willingly inflating your benchmark performance and defrauding new investors / users / customers, I think.

There are no comments on this comment yet.