Because its too computationally expensive. Attacks like these are found when you have access to the model, take an inout image and change its pixel values and see when an anomoly occurs. It’s much faster scoring N images vs. building N models.

How do you combat it? Well that’s an open research question. IMO the most promising techniques make the system harder to attack (ensemble models, more complex models, randomizing the input slightly and dropping outliers) but its not a guarantee. Like in security, it would be great to verify a model is safe to 200 years of brute force search for attack, or whatever it may be.

What about applying dropout to the input image

I don't believe that many people are using K-fold cross validation at all with deep learning, as the computational overhead is massive.

I must be misspeaking then. I'm not suggesting it run during training, but run the classification algorithm X number of times over an input image with chunks areas removed/suppressed from the data.

This is very common. It's an easy way to improve accuracy on a model with a fixed amount of data. They typically crop the image as well as rotate and scale it in order to get a larger data set.

It does make the model more robust, but doesn't seem to help much with finding adversarial examples in the model.

Generating adversarial examples and training on that might be a better approach to solving this.

If I understand correctly, you want to modify an input image at application time to get multiple different classifications, and then compare them to find out how certain the model really is.

While that would likely improve results a bit, it would also multiply the model runtime. That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty.

However, random augmentation can't fully protect against adversarial examples. The number of input variables is simply too large, and there are exponentially many directions in which they could be modified. Data augmentation can't cover all of them, and a single modification that confuses the model slightly can be amplified into an adversarial example that causes a total misclassification.

> That's why the other replies directly jump to talking about training data augmentation, since that can give you similar benefits without the runtime penalty.

Exactly. It's a fix that doesn't work, apparently, so that's why I'm thinking towards the runtime.

> it would also multiply the model runtime.

Predictably so, I would think? Such an approach could scale decently since it's not adding a dimension to the runtime, just a multiple.

Predictably slow is still slow. You could parallelize it over multiple GPUs, but GPUs are expensive, so nobody is going to do that. Requiring lots of training resources is fine, but inference needs to be cheap.

More problematic is that your approach isn't going to actually work, since CNNs are just too flexible (they can learn even completely random labels) and only generalize by accident. No input augmentation technique that doesn't cover every possible modification is going to be robust against adversarial examples, and getting that amount of coverage requires an exponential blowup in runtime. The adversary has the advantage of being able to choose one modification, while the model needs to defend against all of them.

There is a way to do this in Tensorflow, where you try to expand your training data images by altering them, moving them around, etc. However, this is not used with large training sets.

not as massive as producing a bum model. And most of the overhead in ML is data acquisition, cleaning, normalization and tagging costs.

>I'm curious if the attack adds doubt, in that the prediction algorithm is _close_ to truth but gets confused (likelihood of horse slightly less than dog), versus incorrect certitude (the horse is definitely a dog).

While I can't speak for this attack in particular, there exist algorithms that can fool a neural network into generating high-confidence incorrect predictions for images that are visually indistinguishable from ones on which the network performs just fine. That's the biggest issue with these adversarial images.