I never understood the whole notion of attacks on deep-learning systems and all the research effort spent on defending against such attacks. After all, we're talking about systems that are brittle in the face of non-adversarial non-attacks, e.g. http://news.mit.edu/2019/object-recognition-dataset-stumped-....

The language of attacks and defenses implies that we are approaching the kind of robustness that we expect from say a bank app, when in fact we are lightyears away from that.

I'm the lead author of the paper you cited. Glad you enjoyed our work :)

Sure, you can break systems. That doesn't mean that they aren't useful! In many cases a system will see the same boring input many times over. People are often willing to be a bit flexible and help out when it happens to misread something. The fact that you can intentionally break systems like that, and that you can break them in a particular direction, like making them always think there's no danger in an image, is really worrisome.

Our work shows that your autonomous car won't always work well; that its vision system has some systematic error which we can characterize now. Adversarial attacks show that someone can intentionally make your car see a lane, whenever and wherever they feel like it, and drive you off the road. It's a whole different ballgame, and the language of attack and defense really fits well.

Thank @0xab for answering this question. Actually I hold the same opinion as @0xab. Those "non-adversarial non-attack" happens sometimes and there also exist adversarial attacks where adversary intentionally craft some pictures to fool the system. In some cases, the system already achieves a pretty high performance and generalize very well on various datasets but they may fail on adversarial images. For example, attackers can put some carefully-crafted stickers on the STOP sign and the self-driving car will give wrong predictions. The existence of adversarial attacks poses great challenges in applying deep learning model in safety-critical scenarios.

Both non-adversarial non-attack and adversarial attack reflect the vulnerability of the current deep learning models. Thus it is of great significance to study these problem and find countermeasures against those attacks.

Maybe we want the machines think as human think

DeepRobust is a pytorch library for adversarial attack and defense methods on images and graphs. It is demonstrated that some imperceptible perturbation on the input data can fool the systems based on deep learning.

I’ve been working on GCN for a while and I recently read something about the robustness of GCN then I found this. Probably it is useful to get a quick start:)

I think the robustness might be a big issue for safe AI. I’m still trying to learn something related. Do you have any advice where I should start?