The fact that a human isn't fooled by the attack (we can still recognize the 32x32 images for what they are), points to an interesting gap in the abilities of conventional convolutional neural nets.

A better analogy would be stimulating an individual receptor on the retina, in an eye with only 32^2 such receptors. When we see these pictures, we've got a much larger set of inputs to work with.

That's only because the attack is designed to target that particular network. Just wait until we understand real brains better and can generate tailored attacks...

Isn't that basically what an optical illusion is?

Yes, and also this: https://en.wikipedia.org/wiki/Dazzle_camouflage

Hacks human brain rather efficiently.

> Dazzle was adopted by the Admiralty in the UK, and then by the United States Navy, with little evaluation. Each ship's dazzle pattern was unique to avoid making classes of ships instantly recognisable to the enemy. The result was that a profusion of dazzle schemes was tried, and the evidence for their success was at best mixed. So many factors were involved that it was impossible to determine which were important, and whether any of the colour schemes were effective.

It is true that battlefield efficiency of such camouflage is unknown - but I think one can see the effects it does on the brain without conducting a proper rigorous study. The question here is not whether the effect exists - which is IMO obvious - but whether it's enough to make difference in actual combat.

Plus cognitive biases. Those are already well understood and used in a variety of ways.

The one-pixel attack works for pretty much any machine learning model, not just one single neural network.

500,000 years ago our eyes and vision were probably significantly worse than they are today. As our eyes evolved to capture the world better our brains also evolved to correct the errors from our eyes. On the other hand, we feed into our neural networks high quality images. It's true that they are low resolution but they don't contain noticeable noise or artifacts. The attack described here is a smart application of salt and pepper noise. It's ineffective on humans because our vision evolved to filter it out, but a network which has seen only noiseless images is helpless.

I'm curious whether training the network by adding noise and other mutations to the set would make the network more resilient to this attacks. In other words, it's the training set or the network architecture that's vulnerable here?

>I'm curious whether training the network by adding noise and other mutations to the set would make the network more resilient to this attacks. In other words, it's the training set or the network architecture that's vulnerable here?

This is called adversarial training and is currently the most popular technique for protecting neural networks against this type of attack. That being said, it doesn't work as well as one would hope: the adversarially trained models are usually still vulnerable to other attacks.

Not a new concept in general, just a new approach for it with a single pixel change.

There are other concerns of practicality as well, such as dependence on ability to rerun samples through the original network.

The authors of the paper have successfully conducted the attack on 227*227 images as well. The pixels are much harder to see with the human eye.

It seems that larger images increase the search space as a linear function of the dimensions. That is to say, it does take more time to find such pixels, but they are still relatively common.

There are attacks where all pixels are just slightly changed and the classification is completely wrong. Think of changing all pixels by just the least significant bit. Still these changes are invisible to (my) human eye. Basically, each distance function has atttacks with a very small distance.

In the sources of the single pixel paper I found a reference to a prior work which does this: http://www.shivakasiviswanathan.com/CVPR17W.pdf