The fact that we aren't shooting for rotation-invariance is a problem with LFW, not necessarily this algorithm, but you're right to say that the authors' paper would be much more convincing if they tried their approach on several different datasets.
(In particular, the "Multi-PIE" dataset aims to specifically stress rotation-invariance, even though it's not quite as popular as LFW. These guys do use Multi-PIE as part of their training and they do in fact measure their algorithm's performance on Multi-PIE; see page 24. They don't seem to talk about it much since everyone is crazy for LFW and since Multi-PIE is "easy" for other reasons...)
Everyone took the same courses and more or less familiar with the basic algorithms. The task of recognizing a passport-like pictures is nearly trivial, while a "random, on the street" face recognition is a different problem.
People are using environmental and contextual cues (so, does facebook's system) and they mostly guessing rather than "exact matching" peoples.
Anyway, as long as it is not a passport photos the task of face recognition has almost nothing to do with basic NN algorithms, but a lot with contexts and cues.
There is some of that work going on out there, but I would be very surprised to find that spirit in a paper with the words "LFW" in the title. It's just a different focus.
In its time, LFW was "hard" since most of the face datasets (FERET, AT&T) were completely controlled: the subject visited a laboratory, sat down in the best pose, made the best face expression, and had their picture taken by the best camera that grant funding could buy. The "in the wild" part of "Labeled Faces in the Wild" alludes to the fact that these are more real-world than what the community was used to.
It's time to continue that train of thought and move on to something harder though.