At some point we need to trust the counterparty to do the right thing regarding how their product is represented relative to the benchmark. If someone repeatedly puts out deceptive figures the customer should eventually catch on and begin to disregard anything, not just the particular benchmark they happened to exploit.
It's practically impossible to create a benchmark that can't be gamed when the input space to your system is so very large.
What is the alternative? Test all the queries on all datasets?
I find his argument aligns perfectly with my experiences and expectations moving forward. You can't give these reward-seeking models a broad goal that they can solve narrowly and leave them unattended. The problem is that the rigor is being applied so late that it is ruining the system. I disagree that further improving rigor and decision making is the right path forward because it, in this case, places that burden on the party of last resort. It should be the first human in the loop, if not the model itself, that triages these issues.
Peer review isn't intended to be the first and last stage of review. These people are lazy and their actions deplorable.
A system is what it does, unfortunately.