See https://arxiv.org/abs/1610.08401 for model agnostic examples of adversarial attacks.

This is a great paper.. the perturbation map is very reminiscent of psychedelic imagery

Psychedelics come up a lot with nn created images, but this is interesting that under the influence of a perturbation suddenly the classifier starts presenting illogical assumptions

Also similar to how people recollect their own classifiers unfolding or showing bias when considered under the influence of psychedelics

Perhaps there is some deeper analogue with a substance influenced brain's neuronal activity

This is great stuff! I've noticed that the inter-model universality is OK (a little over 50% for one of the universal pertubations), but that's still pretty good!

This is much more interesting, thanks!

You don't necessarily need to know the whole model, but you need accurate enough prediction score outputs for each category. I think if you have that, that's enough to numerically approximate the gradient of the output scores and perform gradient descent.

Someone just published a new attack at ICLR that does not even need confidence scores but works solely on the final decision output of the network and works as well as the best known attacks (that so far required gradients): https://openreview.net/forum?id=SyZI0GWCZ&noteId=SyZI0GWCZ

The algorithm mentioned in the paper doesn't need any information about the internal structure of the image classifier.

Is getting to know the model of the classifier so difficult? I mean, for an API, is it not possible to deduce the internal structure of the system based on the Input-output combination?? Just asking.

If by internal structure gp means the number or structure of the layers, then no, it’s not feasible to deduce this.

If they write regular blog posts or publish the architecture in a journal however, and the api gives you probabilities, you could maybe recreate it with a process similar to distillation?

There are different kinds of attacks. This one, and related ones that minutely change the input, require you to know some details about the model. There are other kinds of attacks, where you don't need to know what the specific model is. For example, http://rocknrollnerd.github.io/ml/2015/05/27/leopard-sofa.ht...

The fundamental problem is that there are only so many training examples and layers that you can fit in a gpu's memory. The current state of the art neural networks work well for very specific tasks (ie handwriting recognition), but don't generalize.

Neural networks do generalise well, and their vulnerability to crafted attacks is unrelated to generalisation anyway.

Additionally the size of the network isn’t the same thing as GPU memory and you don’t fit training examples in memory anyway (?!).