News for hackers. Stuff that matters.
Re:Audio Adversarial Examples by aeling on Tuesday January 30, 2018 @01:34PM (#16267677)
Prior discussion / link to arxiv: https://news.ycombinator.com/item?id=16220376
Re:Audio Adversarial Examples by bcheung on Tuesday January 30, 2018 @05:11PM (#16269576)
There is a similar attack against image classifiers.

So is this just a proof of concept or can this be exploited in the wild?

Based on my understanding you need to have access to network itself (weights, baisses, activation function, architectural topography) to pull off this kind of attack. Doesn't seem like this could be easily be duplicated as an outside agent.

Re:Audio Adversarial Examples by chestervonwinch on Tuesday January 30, 2018 @08:43PM (#16270686)
To generate adversarials, you really just the Jacobian of the network, which, without the network architecture, etc., would require estimation via finite differences which would require running the same signal as input multiple (lots! for large signals) times with minor perturbations to each component. I think whether or not this is feasible depends on the circumstances. Also, I could be wrong about everything because I'm not super current on this stuff.
Re:Audio Adversarial Examples by whataretensors on Wednesday January 31, 2018 @04:10AM (#16272561)
You don't need access to the network, you can just build another one.

https://arxiv.org/abs/1602.02697

"Our attack strategy consists in training a local model to substitute for the target DNN, using inputs synthetically generated by an adversary and labeled by the target DNN"

Re:Audio Adversarial Examples by chestervonwinch on Wednesday January 31, 2018 @06:45AM (#16273143)
that's clever and sneaky!
Re:Audio Adversarial Examples by evjrob on Tuesday January 30, 2018 @05:36PM (#16269729)
I wouldn't be surprised if these speech to text models are susceptible to black box attacks in the same way that image classifiers are: https://arxiv.org/abs/1605.07277
Re:Audio Adversarial Examples by jchw on Tuesday January 30, 2018 @04:31PM (#16269238)
I wonder if it's possible for an attack like this to work against the human brain.
Re:Audio Adversarial Examples by grondilu on Tuesday January 30, 2018 @09:02PM (#16270749)
That's the purpose of magic[1], isn't it? Maybe one day magic tricks will be designed by AI. It'd be funny, since that would gave Arthur C. Clark famous quote ("any sufficiently advanced technology...") a whole new meaning.

1. https://en.wikipedia.org/wiki/Magic_(illusion)

Re:Audio Adversarial Examples by bcheung on Tuesday January 30, 2018 @05:09PM (#16269551)
Optical illusions have a similar effect. Also images that people can see in 2 different ways like the famous old/young woman example.
Re:Audio Adversarial Examples by colechristensen on Tuesday January 30, 2018 @04:56PM (#16269456)
Flashy cartoons give some people seizures.

I'd bet they could be engineered to affect a higher proportion of the population.

Re:Audio Adversarial Examples by taneq on Tuesday January 30, 2018 @08:05PM (#16270502)
I'm pretty sure some alarms use a strobe light at a particular frequency combined with a loud 2kHz tone to discombobulate intruders. That's more sensory overload than an adversarial signal, though.

I'd expect a malicious adversarial example to be more like the ones discussed in Snow Crash or BLIT ( https://en.wikipedia.org/wiki/BLIT_(short_story) )

Re:Audio Adversarial Examples by taneq on Tuesday January 30, 2018 @07:59PM (#16270486)
You could argue that cartoons or line drawings are "adversarial examples" that are reliably identified by the human visual system as representing a particular thing despite being quantitatively nothing like what they're meant to represent.
Re:Audio Adversarial Examples by IIAOPSW on Tuesday January 30, 2018 @11:22PM (#16271276)
Remember the black/white gold/blue dress thing from a while back?
Re:Audio Adversarial Examples by stepik777 on Tuesday January 30, 2018 @04:49PM (#16269389)
What do you mean? How would it look like?
Re:Audio Adversarial Examples by jchw on Tuesday January 30, 2018 @05:24PM (#16269652)
Adding noise to a signal to cause it to be perceived unintuitively. It's possible that, as others pointed out, optical illusions are an example of this. But in actuality, if such a thing existed, I'm not sure we'd be able to tell it's adversarial.
Re:Audio Adversarial Examples by ttul on Tuesday January 30, 2018 @01:15PM (#16267474)
I can’t wait to try this on my friend’s Google phone.
Re:Audio Adversarial Examples by justme22 on Tuesday January 30, 2018 @01:21PM (#16267546)
It won't work on google phone right now, from the paper: "The audio adversarial examples we construct in this paper do not remain adversarial after being played over-the-air, and therefore present a limited real-world threat; however, just as the initial work on image-based adversarial examples did not consider the physical channel and only later was it shown to be possible, we believe further work will be able to produce audio adversarial examples that are effective over-the-air."

However prior work [1] was shown to work on google phones, it is just much more noticeable.

[1] https://www.youtube.com/watch?v=HvZAZFztlO0

Re:Audio Adversarial Examples by Ar-Curunir on Wednesday January 31, 2018 @01:30AM (#16271944)
Just so you know, Nick Carlini was also on that prior work.
Re:Audio Adversarial Examples by aeling on Tuesday January 30, 2018 @01:30PM (#16267645)
I believe this requires the attacker to have access to the ASR neural net's weights, so Mozilla's seems like the only popular framework that's vulnerable right now (not that I'm opposed to them keeping things open).
Re:Audio Adversarial Examples by mbebenita on Tuesday January 30, 2018 @03:19PM (#16268632)
This is exactly why we need to keep ASR tech open. These kinds of issues affect DNNs in general, and other ASR engines based on them are also vulnerable. Having models and the data used to train them open is a great way to help academia make progress in this space.
Re:Audio Adversarial Examples by ladberg on Tuesday January 30, 2018 @04:08PM (#16269037)
Adversarial examples don’t actually have to be generated with the same neural network. If you want to attack another NN, you can make your own with similar training data and similar structure and the attacks can carry over to some extent.
Re:Audio Adversarial Examples by debt on Tuesday January 30, 2018 @04:25PM (#16269191)
Remember WEP encryption? I feel like we're at that stage with NLUI except they mostly have zero security.
Re:Audio Adversarial Examples by svilen_dobrev on Wednesday January 31, 2018 @10:25AM (#16274524)
would this be useful to "massage" a voice phone conversation, so the wannabe wiretappers cannot (automaticaly) decode what really is in it?
Re:Audio Adversarial Examples by jdalgetty on Tuesday January 30, 2018 @02:30PM (#16268194)
None of the examples triggered my google home.
Re:Audio Adversarial Examples by jamesgeck0 on Tuesday January 30, 2018 @04:09PM (#16269047)