I think that working around these type of fooling is easy but not really worthwhile for now. After all, adversarial models are designed to improve the performances of the models.

Also in the article, they test a detector that has to identify a single object in an image that contains two: place an actual toaster next to the banana and call it fooled.