The design of networks is limited by the dataset, since researchers are just trying to hit the metric for the dataset. Better datasets will lead to better networks. Better data eventually means billions of cameras, for example.
The article was kind of interesting to me because it reveals that networks are probably sometimes making decisions on highly discriminating but non-essential stimulus features.
It's like the networks might have a high success rate with large number of replicated examples, over sample size, but not over a large number of distinct examples.
They're overfitting, but in a way that isn't immediately obvious because the test stimuli tend to be limited.
My guess is the answer is to incorporate into the training set a lot of quasi-random or structured but abstract images as controls for training.
The design of the network is a direct consequence of the dataset+metric because researchers focus on accuracy scores against the test data. Given a better dataset and metric, researchers will solve it with a better network design. I guarantee it.
But we don't actually know the real issue, do we? You can't easily debug a neural network. Sure, you can throw ten times more data at it and hope that fixes it. But the more data you throw in, the less you understand what's happening - which affects your ability to anticipate the next problem.
Improvements in datasets, transfer learning, and online learning will help. Unfortunately this underscores the issue that giants such as google have more pictures of funny glasses than anyone else...