Augmenting your training dataset with adversarial examples is known as adversarial training, see e.g. [0] for a recent overview with empirical results. This seems to be a good first step in defending against such attacks, though the most naive approach of adversarial training doesn't work as well as you'd expect.

[0] https://openreview.net/forum?id=rkZvSe-RZ