I definitely trust the totally private dataset more.

(FWIW, and not directly related to my question, I always thought of GPL as the less (not more) restrictive license from the perspective of a user, because I could always ask for the source code and debug a problem on my own.)

Older, non-instruction tuned models needed post-training on public datasets to even reliably produce meaningful answers.

Now we're testing tasks that are so complex that the LLM should reasonably be expected to answer without additional post-training.

Once you have a public dataset, even feeding those examples to an LLM and producing synthetic variations is enough to let you game the benchmark. And the worst part is you don't need to be unethical to do this: some people would say it's just a good way to expand your training data even though it incidentally allows you to overfit on the task, without overfitting on the public dataset.

So everyone's doing stuff like that, and we're getting models that are increasing overfit to a few narrow tasks.

-

The alternative is just giving detailed plain english descriptions of the tasks in question. Those can be used to generate synthetic tasks, but won't result in matching the benchmark's "shape" perfectly (as long as the questions stay hidden), and that alone is enough to ensure some level of generalization takes place.

It looks like one court did in a non-precedent binding case, but I might be remembering incorrectly.