The problem raised in the paper is that most of the time, you can safely assume that if you have a new data point that is very close to a bunch of correctly-classified data points and far away from points of different classes, the new data point will be correctly classified. In other words, you assume the classification probability is locally smooth.

The problem is that the adversarial examples they are able to come up with are very close to the original images, so this means the smoothness assumption seem to be invalid for deep learning models. As they put it in the paper : "Our main result is that for deep neural networks, the smoothness assumption that underlies many kernel methods does not hold."

It's going to be interesting to see what happen when other researchers try to replicate the results for other models and datasets.