News for hackers. Stuff that matters.
Re:How Adversarial Attacks Work by munificent on Thursday November 02, 2017 @03:32PM (#15613495)
I think your logic is:

1. All ML algorithms can be fooled trivially. 2. The human mind cannot be fooled trivially. 3. Therefore, the human mind is not an ML algorithm.

But claim number 2 is clearly wrong. Human minds are trivially fooled. Here's one:

http://www.jimonlight.com/wp-content/uploads/2012/02/Paralle...

This is exactly what an optical illusion is.

Re:How Adversarial Attacks Work by schoen on Thursday November 02, 2017 @03:37PM (#15613525)
Yes, when I read "Humans have no problem correctly interpreting adversarial examples, so there must be a way to do it automatically" in the article I thought "aren't optical illusions adversarial examples for human perception?" and also "maybe we just haven't found the other kinds of adversarial examples that work against human perception because we can't show humans sufficiently large numbers of examples and measure the humans' responses accurately enough".

But the optical illusion example seems very strong because if you say that human perception includes abilities to classify things, there are many examples just like your link that show cases where the classification routinely goes wrong.

It seems possible to me that there are as-yet undiscovered adversarial examples in human perception that we simply don't have any feasible way to search for, and may never be able to construct practical examples of. (It would probably be extremely unnerving to experience one in real life.)

Re:How Adversarial Attacks Work by yters on Thursday November 02, 2017 @04:01PM (#15613724)
Optical illusions are reverse adversarial examples to those who think the human mind is a deep neural network.
Re:How Adversarial Attacks Work by danharaj on Thursday November 02, 2017 @03:43PM (#15613586)
Cool illusion; I figured it out and so will most humans. I don't think the claim that these sorts of illusions are of the same nature as, say, single pixel attacks against neural networks is justified. On the other hand, it is also true that it's not clear whether such attacks on our current crop of primitive neural networks will work on more advanced autonomous systems that regulate themselves.
Re:How Adversarial Attacks Work by yters on Thursday November 02, 2017 @04:04PM (#15613742)
In theory it seems that with sufficient understanding of any ML system, regardless of sophistication, such examples can be constructed. It is essentially the brain in the vat problem. Computers have no access to objective reality, whereas we cannot say the same regarding the human mind without making strong assumptions about the nature of the human mind.

To put this in comp sci language, we can think of DNNs as proofs that certain inputs belong to certain classes. Since no proof system can be both complete and consistent, something outside the proof system can always either provide unprovable examples or adversarial examples.

Re:How Adversarial Attacks Work by danharaj on Thursday November 02, 2017 @04:42PM (#15614112)
I think the proper test is how stable the learning system is in the face of adversarial examples. I have no indications either way that we could, using current ideas, create a system that learns and is autonomous like a human brain while regulating itself from going too far off the rails. An optical illusion won't lead to you thinking you're the king of France, nor do there seem to be basilisks that will actually crash a brain.

It remains to be seen if the same holds for artificial learning systems. Of course! The gap between current learning systems and even the brain of a rodent is vast and we have yet no idea how to get there.

Re:How Adversarial Attacks Work by munificent on Thursday November 02, 2017 @05:27PM (#15614441)
> An optical illusion won't lead to you thinking you're the king of France

I'm not sure what you're getting at here. By definition, optical illusions are perceptual mistakes limited to the optical system.

It is certainly possible for an optical illusion to trick someone into doing something they don't intend to do because they are fooled into believing the illusion. Imagine painting a set of stairs with a disorienting pattern of shading that makes them appear off, leading you to trip and fall down them.

I don't think it's necessarily for a perceptual illusion to hijack your entire belief system in order to be considered an effective adversarial attack.

Re:How Adversarial Attacks Work by yters on Thursday November 02, 2017 @05:49PM (#15614594)
The whole point of these examples is the DNN gives erroneous classifications with near complete certainty. This looks to me like hijacking a belief system.
Re:How Adversarial Attacks Work by yters on Thursday November 02, 2017 @05:12PM (#15614330)
The incompleteness/inconsistency theorem I mention seems to indicate no such ML system is possible. It will either encounter examples it cannot deal with, or it will be deceived. ML can only be guaranteed to work within very limited domains.
Re:How Adversarial Attacks Work by YeGoblynQueenne on Thursday November 02, 2017 @04:32PM (#15614013)
The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only.

It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that.

Note that I don't agree that this tells us anything about whether the human brain (or mind) is "like" a machine learning algorithm. To me this question has about as much meaning as asking if the brain is "like" quicksort.

The physical substrates are so clearly different that the only comparison you can make is on the level of capabilities (say, both are Turing-equivalent etc) not that of actual structures. Like, where's the 1's and 0's in the brain?

Re:How Adversarial Attacks Work by munificent on Thursday November 02, 2017 @05:19PM (#15614382)
> It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that.

Here's a physical object that makes you mistake an insect for a plant:

https://en.wikipedia.org/wiki/Phasmatodea

Re:How Adversarial Attacks Work by romaniv on Thursday November 02, 2017 @06:08PM (#15614756)
Problem is, mimicry involves copying essential properties of the target object: color, texture, shape, movement dynamics and so on. It relies on true ambiguity. Adversarial examples against neural networks (the interesting ones, anyway) involve a combination of insignificant, seemingly random permutations that only work in their totality. That's a very important difference.
Re:How Adversarial Attacks Work by munificent on Monday November 06, 2017 @12:25PM (#15637023)
> essential properties of the target object

They are only "essential" properties according to the limitations of your particular perceptual system. To a bee that can see UV light, a stick bug may look entirely different from an actual stick. An animal that hunts by scent would find them clearly distinct.

There is no such thing as "true" ambiguity unless the two objects are actually the same thing in all respects. If they are distinct but appear the same, it's because they are overlapping in some respects but not all.

Re:How Adversarial Attacks Work by romaniv on Tuesday November 07, 2017 @01:40PM (#15646178)
> They are only "essential" properties according to the limitations of your particular perceptual system. To a bee that can see UV light, a stick bug may look entirely different from an actual stick. An animal that hunts by scent would find them clearly distinct.

This is a very confused statement. Fist, since we are talking about image recognition we are - by definition - talking about vision in the spectrum that can be captured by a digital camera and encoded in a typical image format. Second, there definitely is such a thing as essential property. It is a matter of correlation with reality, as well as internal consistency. For example, plants are green and have leaves because of the way they use sunlight. So permuting color of all leaves in the picture is fundamentally different from permuting luminosity of some random pixels.

Re:How Adversarial Attacks Work by YeGoblynQueenne on Thursday November 02, 2017 @06:48PM (#15615050)
What romaniv said- it's a different thing.
Re:How Adversarial Attacks Work by chiaro on Friday November 03, 2017 @01:42AM (#15616922)
A more accurate example in the non-2D realm: https://en.wikipedia.org/wiki/Pareidolia
Re:How Adversarial Attacks Work by throwaway613834 on Thursday November 02, 2017 @05:04PM (#15614276)
> It's really hard to imagine an optical illusion that makes you mistake objects in the physical world for something else- say, panda for a lawn mower or a car for a pigeon, or something like that.

Sure but people do, for example, mistake each others' faces or voices. You don't need to mistake your friend for a lawnmower for it to be dangerous.

Also, for example, I often mishear my own name when someone else is speaking. Might not happen with every name but it does with some.

Re:How Adversarial Attacks Work by YeGoblynQueenne on Thursday November 02, 2017 @06:57PM (#15615121)
Well, people misidentify others when viewing conditions are poor or when they dont' know the other person well. For instance, say my mother was standing in front of me in broad daylight and I was given half a second to look at her face; I really don't see how I'd fail to identify here, unless her face had changed drastically for some reason.

So, to be fair, this too is a different thing than what we're talking about.

Re:How Adversarial Attacks Work by jpttsn on Thursday November 02, 2017 @05:51PM (#15614613)
Re:How Adversarial Attacks Work by YeGoblynQueenne on Thursday November 02, 2017 @06:47PM (#15615032)
Again, this is an optical illusion that fools depth perception. It doesn't make you see the T-Rex as a tea-pot, say.

Different things.

Re:How Adversarial Attacks Work by jpttsn on Friday November 03, 2017 @07:17AM (#15618242)
Fair point; it's not exactly the same mistake. If humans made those, this discussion would probably never happen.

What I tried to rebut is your first, weaker assertion:

> The problem with optical illusions like that is that they are, in their vast majority, made of abstract shapes. Most of them play with our perception of distance and depth - and the majority again work on two dimensions, only.

This makes me see a moving 3d object when I am in fact looking at a static flat object; and I can't shake it, even if I try.

Re:How Adversarial Attacks Work by YeGoblynQueenne on Friday November 03, 2017 @01:44PM (#15620939)
OK, but I did say "the (vast) majority".

Also, from what I can tell, this just happens to work better with a meaningful shape (the image of a T-Rex). It has something to do with how we confuse convex with concave shapes and the T-Rex's snout just happens to be very convenient to demonstrate this confusion (e.g., I think an image of a horse would work as well). I think the illusion would also work with an abstract shape- except it would be harder to find a good abstract shape to demonstrate it as clearly as with the T-Rex.

Re:How Adversarial Attacks Work by empath75 on Thursday November 02, 2017 @07:59PM (#15615510)
I know fmri isn't high enough resolution for this, but I wonder if you had real time access to brain state, you could construct adversarial examples like these.
Re:How Adversarial Attacks Work by romaniv on Thursday November 02, 2017 @06:25PM (#15614865)
The comparison in disingenuous. This illusion has nothing to do with image classification and heavily relies on adding misleading context to the object to alter the perception of one of its properties. Most optical illusions are like that. That is not what is being done with adversarial examples.
Re:How Adversarial Attacks Work by yters on Thursday November 02, 2017 @03:57PM (#15613693)
In order for optical illusions to be a counter example, I must believe those lines are not parallel with near certainty. While I perceive them as such, I don't believe they are. The fact that I can know my perception does not match reality is what gives optical illusions their novelty. Thus, the very nature of optical illusions makes them not a counter example.

A true counter example must be similar to making me believe a car is a toaster by manipulating a single cone cell in my eye. Such a possibility is absurd, which leads me to conclude the human mind is not ML.

Re:How Adversarial Attacks Work by red75prime on Friday November 03, 2017 @12:36AM (#15616730)
> the human mind is not ML

Pooling layers in CNNs throw away lots of information about spatial co-occurrence of features, which leads to a possibility of adversarial images where adversarial features are scattered all over the place and so they don't significantly affect humans' visual processing.

The conclusion should be "Human mind is not that kind of ML".

Re:How Adversarial Attacks Work by yters on Friday November 03, 2017 @01:10AM (#15616837)
CNNs are the best we have. If they can't deal with adversarial examples, then what can?
Re:How Adversarial Attacks Work by red75prime on Friday November 03, 2017 @01:18AM (#15616867)
Hinton's "capsule networks", probably. Or any other future architecture which preserves positional information better.
Re:How Adversarial Attacks Work by yters on Friday November 03, 2017 @01:33AM (#15616900)
It is a problem inherent in any ML method with large VC dimension. The model will overfit, and when it overfits, adversarial examples become possible.

The very nature of ML seems incapable of dealing with this dilemma. You either work within a very constrained domain (low VC) to ensure everything is covered, and thus are unable to fit complex datasets, or expand the domain (large VC) and thus overfit the datasets. It's the bias-variance tradeoff, and is insurmountable.

Re:How Adversarial Attacks Work by red75prime on Friday November 03, 2017 @02:37AM (#15617087)
Adversarial examples are no longer adversarial in any meaningful sense, when humans are fooled by them too. We don't need to surmount BV tradeoff, we need to make sure that ML models make the same tradeoffs as humans.
Re:How Adversarial Attacks Work by yters on Friday November 03, 2017 @02:49AM (#15617128)
Evidence that humans are fooled by adversarial examples? Optical illusions and other perception issues don't count, as explained by many others in this set of threads.
Re:How Adversarial Attacks Work by red75prime on Friday November 03, 2017 @03:16AM (#15617215)
Camouflage. Paintings perceived as real 3d objects for a fraction of second.
Re:How Adversarial Attacks Work by yters on Friday November 03, 2017 @06:36PM (#15622649)
You think these are equivalent to changing a single pixel and causing a complete misperception of the object which is believed with almost 100% certainty?
Re:How Adversarial Attacks Work by red75prime on Sunday November 05, 2017 @05:38AM (#15628906)
Single pixel attack is an artifact of the network architecture. It is not inherent vulnerability of every ML technique.