Thanks. But they still need to be able to use the black box a brute force number of times on shady inputs and get detailed outputs during their gradient descent. Not going to be allowed on a check reader or anything sensitive.

Not necessarily. Adversarial examples have been shown to, for instance, be transferable across different networks with different hyperparameters (e.g., number of layers) trained on disjoint subsets of a training set [0, section 4.2]. There are more references from the paper linked by the OP.

[0] https://arxiv.org/abs/1312.6199

Thanks. I wonder if adversarial training helps prevent overfitting too. Could you use adversarial training to beat alphago ?

You could not, because AlphaGo is not a classifier (so it isn't well-defined what an adversarial example is) and the input space is discrete (Go board state) and you can't do ε-small perturbations (two different states differ by at least one stone).