News for hackers. Stuff that matters.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by anishathalye on Wednesday November 01, 2017 @10:44AM (#15601566)
Hi HN! I'm one of the researchers that produced this result: we figured out how to make 3D adversarial objects (currently fabricated using full-color 3D printing technology) that consistently fool neural networks in the physical world. Basically, we have an algorithm that can take any 3D object and perturb it so it tricks a classifier into thinking it's something else (for any given target class).

Recently, there's been some debate about whether or not adversarial examples are a problem in the real world, and our research shows that this is a real concern, at least with current neural network architectures (nobody has managed to solve the problem of white-box adversarial examples yet).

I'm happy to answer any questions that anyone has!

Re:Fooling Neural Networks in the Physical World with 3D Adv... by etiam on Wednesday November 01, 2017 @04:30PM (#15605190)
This was a wondrous read. I had not expected that these things would be significant in real world context with free viewpoint.

Have you looked at whether the perturbation tend to do something to other networks than it's optimized for? If I ran for example VGG-19 and Resnet-50 alongside Inception V3, could I still be reasonably sure to get a correct best-of-three vote if I point the camera on your models?

Do you think your algorithm would support optimizing to fool several known networks in the same way simultaneously?

Re:Fooling Neural Networks in the Physical World with 3D Adv... by andrewilyas on Wednesday November 01, 2017 @04:53PM (#15605383)
What we propose in our paper is a white-box attack, so given access to the classifier gradient, we can make adversarial examples. We don't investigate the "transferability" of our adversarial examples to other networks, though other work in this area has explored that problem (though, of course, not in the EOT / 3D adversarial example case).

I expect it wouldn't be difficult to defeat an ensemble, though (e.g. if you're averaging predictions, you can just differentiate through the averaging).

Re:Fooling Neural Networks in the Physical World with 3D Adv... by etiam on Wednesday November 01, 2017 @05:52PM (#15605817)
Apologies if I bring up things that can be found in the paper at a more careful read, but I probably won't have time to go through it properly before this HN post goes cold. I think I have fairly good notion of what you're doing though.

I share your expectations for an averaging ensemble, which is after all still for most purposes a single model, but let's say I'm concerned precisely about people trying to fool my networks like this and one of the things I do is check the consistency of answers between different models and if they mismatch over a certain threshold I might flag that for an extra check by releasing the hounds or something.

In that context I think it's of interest how the perturbation features you develop affect networks created with similar technology but different choices of architecture and hyperparameters. Are the foreign pertubations neutral there or do they have an effect? If there is an effect, to what extent is it consistent? To what extant can they be superposed in a way that is manageable for getting predictable results for different networks simultaneously? What fraction of the available texture area do you need to affect to get a reliable misclassification, and what is the 'perturbation capacity' of the available area? That last one I think is particularly interesting in your case where presumably you put much more constraint on the texture by requiring that it works for multiple viewpoints.

I totally respect if you, or indeed anyone, can't answer those questions yet, because of focus and stage of research. Personally I have only followed adversarial attacks very superficially so far, because IMO before what you just released it was a point of concern for the mechanics of the ANNs (and inspiration for some good ideas) but for practical purposes more of a curiosity than a demonstrated real concern in applications. (If you're allowed to show people deceptively crafted scenes from exactly the right perspective point they fail too. Just look at Ames rooms. But good luck making that into a significant real-world exploit on humans.)

Any publications you'd care to recommend in the transferability subfield?

Re:Fooling Neural Networks in the Physical World with 3D Adv... by tMcGrath on Thursday November 02, 2017 @03:57AM (#15608604)
I agree - it's a surprising and cool paper. There has been some work done on fooling network ensembles by constructing constructing a Bayesian posterior over weights using dropout [0]. This is an ensemble of weights for the same network, not over different architectures, however.

The basic idea here is that most of the time, each member of the ensemble will misclassify the adversarial example in a different way. This means that the posterior predictive distribution for adversarial examples ends up much broader, and you can detect them this way.

Surprisingly, even this can be beaten in the white-box case [1], although it's by far the hardest to beat of the current adversarial defences, and needs much more distortion. It's beaten exactly as the GP says, by differentiating through the averaging. AFAIK no-one's tried architecture ensembling, but I expect it would be vulnerable to the same technique.

[0] https://arxiv.org/abs/1703.00410 [1] https://arxiv.org/abs/1705.07263

Re:Fooling Neural Networks in the Physical World with 3D Adv... by _0ffh on Wednesday November 01, 2017 @11:18AM (#15601983)
Amazing results! To be fair, the baseball looks like crema or coffee stained milk foam even to me... ;-)
Re:Fooling Neural Networks in the Physical World with 3D Adv... by anishathalye on Wednesday November 01, 2017 @02:54PM (#15604393)
Thanks!

The unperturbed baseball also looks like that -- the model we found is of a dirty baseball :P

We probably should have included a photo of that in our blog post, but you can see our paper if you're interested in seeing what the original looks like.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by mesozoic on Wednesday November 01, 2017 @08:37PM (#15606844)
Awesome can I buy an adversarial tshirt?
Re:Fooling Neural Networks in the Physical World with 3D Adv... by netsharc on Thursday November 02, 2017 @06:34AM (#15609180)
It would be a neat trick to fool face scanners, for bonus points the shirt should just have a face of Thomas Crown: https://www.youtube.com/watch?v=C3rDWENRI7c

Or I guess it's more effective to fool the scanners to think you're a dog/cat instead of a potential terrorist (because everyone is one nowadays, right NSA?).

Re:Fooling Neural Networks in the Physical World with 3D Adv... by flavio81 on Thursday November 02, 2017 @10:37AM (#15610761)
Congratulations to you and your team for inventing what i'd call Neural Network Trolling (NNT), and thus giving us humans a way to escape the tyranny of automated decision systems.

The consequences for the future can be enormous, seriously.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by UncleMeat on Thursday November 02, 2017 @12:19PM (#15611762)
White box adversarial examples are not new. The novelty here is producing physical objects that behave like adversarial examples.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by darkmighty on Wednesday November 01, 2017 @03:00PM (#15604447)
Hi there!

What do you think are currently the best method to fight adversarial weakness?

Would simply generating an augmented training set including adversarial examples be sufficient? (in your case, for example, you would include random poses of the adversarial 3d models in the training set) Or do you think a totally new architecture of training method is necessary to deal with it?

And if it's the latter, do you think this is a symptom of a more general shortcoming of current architectures, or a localized issue?

Thanks, great work btw!

Re:Fooling Neural Networks in the Physical World with 3D Adv... by logane on Wednesday November 01, 2017 @03:37PM (#15604775)
Another author here - adversarial training is not sufficient to protect against white-box attacks but it seems to be the best method we have so far (https://arxiv.org/abs/1706.06083).

It seems that all current state of the art architectures are vulnerable to adversarial examples; to the best of my knowledge there are no image classification networks that researchers have failed to reliably produce adversarial examples for.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by rota219 on Wednesday November 01, 2017 @11:31PM (#15607625)
The best method for neural networks robust to l_infinity bounded iterative adversarial attacks is this https://openreview.net/forum?id=S18Su--CW (although it is still currently under review).
Re:Fooling Neural Networks in the Physical World with 3D Adv... by darkmighty on Thursday November 02, 2017 @12:38PM (#15611965)
Looks like a very nice paper, thanks.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by darkmighty on Wednesday November 01, 2017 @09:11PM (#15607042)
I see. I ask because there's a common observation that a human (and perhaps an AGI) would never really even run the risk of confusing say a turtle with a riffle, which I tend to agree with.

Part of it is mistaking the forest for the trees. It may be just an artifact from the requirements we place on image classifiers (which are very lax) and the way we train them, not anything fundamental.

Indeed I believe we tend to think with more solid logic, specially when the decision becomes difficult. A DNN will look at the statistics of a feature set and make judgement upon that. A human can categorically reject certain hypothesis from definition requirements: a Riffle is a weapon. It must have a barrel to guide the projectile, and a muzzle for it to exit. It must have some firing mechanism (usually a trigger). Even if at a glance we get confused by exactly what an image is picturing, we can make quick logical judgements on sub-features to make epsilon-misclassifications almost impossible.

A network that would act that way would need some recursive behavior (to implement the varialble-time classification efficiently), a recursive "logic module" or "language module" plugged into the end of naive feature classification.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by readams on Wednesday November 01, 2017 @09:51PM (#15607202)
You're assuming we couldn't make adversarial examples with white box access to your brain. It's entirely possible that such examples do exist.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by red75prime on Thursday November 02, 2017 @01:17AM (#15608021)
We don't need white box access: https://i.imgur.com/mOTHgnfl.jpg
Re:Fooling Neural Networks in the Physical World with 3D Adv... by darkmighty on Thursday November 02, 2017 @09:47AM (#15610342)
That's an adversarial picture, but not nearly as dramatic as the epsilon-adversarial examples -- and in particular there are no logical inconsistencies (confusing a cat for guacamole or a turtle for a riffle). I don't think anyone doubts a form of camouflage or isomorphic deception is unavoidable.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by darkmighty on Wednesday November 01, 2017 @10:50PM (#15607458)
I don't doubt there are weaknesses, or even fundamental lack of interpretability for classification spaces that overlap (i.e. have a "morphing sequence"). One example that sticks out from my childhood is this image:

What do you see here?

https://seeklogo.com/images/A/Antarctica-logo-9202568406-see...

My entire childhood I saw a weird face (without thinking too much about it). This is the logo of a brazilian beer brand. When I was a teenager I saw an ad with two penguins, and then it clicked.

Optical illusions are well documented too, some classic examples:

http://www.optics4kids.org/osa.o4k/media/optics4kids/womanil...

https://obasandbox.files.wordpress.com/2012/07/carolines-vas...

http://i.telegraph.co.uk/multimedia/archive/01120/blackballs...

But it's quite probable we'd found any glaring adversarial issues by now. After all artists can conduct a semi-whitebox, mostly blackbox adversarial optimization of illusions (I'm sure some process like this is how they came up with the Old/Young lady illusion). Note however that even those are particular errors like incorrect brightness estimation, or near-complete dichotomy (it's not that either the young or old interpretation are incorrect in some sense). An epsilon-failure seems much more difficult to come up with. The distinction seems mostly in the ability to apply basic logic on top of pure pattern recognition, sort of greatly enhancing the decision boundaries through recursive thought. Eventually logical features (a quick "proof" of sorts) win. You "prove" that what you're seeing indeed can't be a rifle, it must be a weird turtle.

This logical approach probably has gradations in power. In general it might even be algorithmically undecidable whether an image is logically valid. Generalizations of Escher illusions:

https://upload.wikimedia.org/wikipedia/en/e/e8/Escher_Waterf...

come to mind. In practice we cut off the decision process when we've found main logical features and connections, so unless the image inspires superficial uncertainty a deep logical inconsistency could slip by (that a more intelligent person may not miss by universally applying a deeper consistency check).

Re:Fooling Neural Networks in the Physical World with 3D Adv... by highd on Wednesday November 01, 2017 @03:54PM (#15604885)
How are you optimizing over textures? Do you have a differentiable renderer? Or are you not calculating those gradients via backpropagation?
Re:Fooling Neural Networks in the Physical World with 3D Adv... by anishathalye on Wednesday November 01, 2017 @05:06PM (#15605467)
Yeah, we implemented a differentiable renderer.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by highd on Wednesday November 01, 2017 @07:05PM (#15606301)
Please release it! There are a lot of applications for a differentiable renderer that's compatible with DNN libraries.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by anishathalye on Wednesday November 01, 2017 @07:12PM (#15606349)
Yeah, totally agree that it could be useful to the community -- open-sourcing it under a permissive license is on our todo list :)

It's a reasonable amount of effort to disentangle it from the rest of our code and build a nice API, so it will be several weeks till we get around to this.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by blt on Wednesday November 01, 2017 @08:13PM (#15606734)
please do, this could be a huge contribution to the community.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by aaronsnoswell on Wednesday November 01, 2017 @07:53PM (#15606609)
A very interesting result with big implications for the future AI arms race :) Congratulations and thanks for sharing!
Re:Fooling Neural Networks in the Physical World with 3D Adv... by vtange on Wednesday November 01, 2017 @02:43PM (#15604259)
Do you know how exactly the distorted object differs from the original? Is it based on color? lighting? Some subtle difference in texture or material?

I know that for 2D images there currently exists something like this where some sort of pixelated noise (color?) is added to the image to force an incorrect classification.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by anishathalye on Wednesday November 01, 2017 @02:53PM (#15604389)
We only modify the texture; we retain the original shape of the object.

If you look closely and compare the unperturbed original with the rifle turtle, you can spot some differences.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by cwkoss on Wednesday November 01, 2017 @04:23PM (#15605123)
If the adversarial turtle is far enough from the camera that it cannot discriminate the detail of the texture, does it get recognized as a turtle?
Re:Fooling Neural Networks in the Physical World with 3D Adv... by fudged71 on Wednesday November 01, 2017 @03:04PM (#15604488)
Was it an MCor printer? Which prints did you use to correct for color artifacts in the print process?
Re:Fooling Neural Networks in the Physical World with 3D Adv... by anishathalye on Wednesday November 01, 2017 @03:15PM (#15604575)
We got all our prints through zverse.com, and we used full color CYMK printing.

We didn't correct for color artifacts; we just used a wide distribution with EOT (we just modeled color inaccuracy rather than trying to make a color map or something).

Re:Fooling Neural Networks in the Physical World with 3D Adv... by hooloovoo_zoo on Wednesday November 01, 2017 @07:37PM (#15606515)
If you added the y-axis from your cat figures to your turtle videos, what would we see?
Re:Fooling Neural Networks in the Physical World with 3D Adv... by hinkley on Wednesday November 01, 2017 @07:25PM (#15606435)
What is your team mascot and why is it Winnie the Pooh dressed as a rain cloud?
Re:Fooling Neural Networks in the Physical World with 3D Adv... by deepthrowaway on Wednesday November 01, 2017 @05:19PM (#15605558)
The inception network tested is not trained for multilabel classification problem, I believe it would be harder to function against a multi label network such as inception trained on Open Images dataset.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by j2kun on Wednesday November 01, 2017 @04:09PM (#15605013)
What "high risk" real world systems use deep learning?
Re:Fooling Neural Networks in the Physical World with 3D Adv... by Rotten194 on Wednesday November 01, 2017 @04:31PM (#15605195)
Maybe self-driving car vision systems? I'm not sure if this is how they're actually implemented, but it would make sense to me.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by j2kun on Wednesday November 01, 2017 @04:51PM (#15605365)
Well the article asserts this, so I'm asking for clarification on what systems they were thinking of.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by mabynogy on Wednesday November 01, 2017 @02:25PM (#15604075)
I wonder why the network picks a riffle.

Do you have any explanation about that?

Re:Fooling Neural Networks in the Physical World with 3D Adv... by anishathalye on Wednesday November 01, 2017 @02:52PM (#15604372)
It's a targeted adversarial example, we can do this for _any_ target class. We chose for our demo example to turn the turtle into a rifle, which was chosen uniformly at random from the 1000 ImageNet classes.

In our paper, we have many more examples with different target classes, all chosen uniformly at random.

Re:Fooling Neural Networks in the Physical World with 3D Adv... by mabynogy on Wednesday November 01, 2017 @04:51PM (#15605359)
OK. Many thanks for the details.
Re:Fooling Neural Networks in the Physical World with 3D Adv... by zitterbewegung on Wednesday November 01, 2017 @05:53PM (#15605836)
Have you tried a black box attack using transfer learning on a production system ?