Honestly, that abstract makes me more upset. If these are due to NN's nature as linear classifers, then we are all in trouble, given that almost everything useful is based off linear models. Given the title of the paper though, I should probably be more hopeful :)