Thanks for the comment and for adding examples, and for your nuanced comments of the answer overlap split.
My position is that these datasets are still useful for QA, but what was lacking was an analysis of how easy/hard the questions in them were, and what kind of modelling was needed to do well. These overlap phenomena are less like "bugs" maybe, but more like poorly understood features.
We need models that can accurately recall QA pairs they have seen before, so being able to score well on "memorizable" QA pairs is still important to do well, but we also want models that can do more than that. One single accuracy number on a leaderboard cannot capture all the behavioural information we need to properly understand the capabilities of these models.
The paper also considers answer overlap; that is, when the answer to the question also occurs in the training set. This alone does not imply memorization, but it does open the door for some shortcuts. Some examples from the paper are:
It's less obvious how to treat answer overlap. In particular, removing answer overlap might bias the dataset towards harder questions. Some of the reduction in model scores is presumably because models have to work harder to understand the question, as merely looking for topical similarities will be ineffective, but it also removes questions whose answers are general enough to apply to many questions, like ‘8’, ‘the 1930s’, ‘Harvard’, ‘2011’, etc. This means that reductions in the score don't clearly say how much cheating happened. However, the success of the BERT-based Nearest Neighbor model, which retrieves the answer of the most semantically similar fine-tuning sample, scores similarly to BART, which seems much too high for comfort.It should be noted that a lot of the discussions about causal knowledge and intelligence aren't too relevant to these questions, as they are largely tests of memory or retrieval. It is expected that these models answer the questions by searching their training data or their document index. The issue is that their training data isn't meant to contain copies of the test questions, just the information sufficient to answer them.