> We view this gap as the result of a small distribution shift between the original CIFAR-10 dataset and our new test set. The fact that this gap is large, affects all models, and occurs despite our efforts to replicate the CIFAR-10 creation process is concerning.
> Nevertheless, the accuracy of all models drops by 4 - 15% and the relative increase in error rates is up to 3×. This indicates that current CIFAR-10 classifiers have difficulty generalizing to natural variations in image data.
It remains to be seen if others can think up ways to maintain performance that the authors did not manage.
Those models don't seem to be THAT brittle.
I think this is important point to make and talks to the real life limitations of run of the mill SL techniques. If you can't validate with noisy "real world" data, then you're basically staying inside the box and hoping that you've curated well enough to mimic real world conditions.
I still like RL system for the improvement on this, however it's much more difficult in practice.
"Do CIFAR-10 Classifiers Generalize to CIFAR-10?" - https://arxiv.org/abs/1806.00451
They use the same procedure used to construct CIFAR-10 to construct a new test set and test a bunch of state of the art results on the new test set.
They see a generalization gap, but the relative order of SoTA results remains (roughly) the same.
So, yes, test set validation is an overestimate of real-world performance of these systems, but progress on test sets is indicative of progress in real-world settings.
And to remember this in context, no-one is asking "do traditional computer vision systems understand images", because they were explicitly just looking at image statistics.