Couldn't you fix this by running an adversarial transfer learning step after you've trained your model? The adversarial transfer is not going to preserve subtle nuances like this because they should be indistinguishable from regular noise. I'm not a deep learning expert though so I'm sure I missed something subtle.