I recently saw a paper (which I reposted here on HN a while back) that perhaps could help with your problem. The paper proposes an approach that sort of induces models to learn representations of objects that are good at predicting "the most agreed-upon" rotations of the inputs. I'm not explaining it well. Anyway, a model in the paper achieved SOTA on a change-of-viewpoint dataset with a really tiny number of parameters. Might be worth a look: https://arxiv.org/abs/1911.00792

I have been looking for an excuse to use capsule networks. I can't investigate it as a possible solution at work because even if good results are achieved the performance problems[0] would prevent real world use. Definitely might look into that as a side project though. I have also been looking for an excuse to do something beyond like SENet in Julia.

Maybe a new benchmark like this is what we need to get out of the rut.

[0]: delivery.acm.org/10.1145/3330000/3321441/p177-Barham.pdf

I'm the author of that paper. Happy to answer questions about it here.