That's already known: there are inputs to the network that do not make sense and yet will trigger strong responses. Think of them as inputs that have the same effect on NNs that optical illusions have on the human brain. We infer something that isn't there.

I suspect that as network architectures get better and parameter counts drop these will get harder and harder to construct.