I agree that openai is somewhat sketchy about this, but they're sketchy about everything. In the past though they have admitted up front to data contamination (e.g. the original gpt-4 press release did not use big-bench as a benchmark due to data contamination). For the Putnam in particular: this is not a benchmark that they use. There is no reason to exclude it since it is not part of the "test set" in any meaningful sense.