Interesting, thank you for sharing that. It reminds me of this approach called "PIRL" by Facebook AI. They framed the problem to learn invariant representations. You might find it interesting.

https://amitness.com/2020/03/illustrated-pirl/