> Meanwhile, "it's funny how our image classifiers grossly misperform if you rotate the images a little or add some noise."

Same thing arguably happens with humans with rotation. Our eyes even rotate in the roll axis to keep gravity aligned things upright. Most people can draw faces more accurately when copying from an upside down face than a right side up one.