Then, use a recurrent ConvNet at the end to pump out probabilities for each class (in this case face) with a low/tractable number of parameters.

What would pairs of maximally distant faces look like?

What would the mediods of a large dataset that was clustered into two clusters look like? Would it be a high level feature that we'd recognize like male or female?

Would faces that were mediods of the cluster be considered more attractive?

Would it be possible to build a similar app that allowed you to identify an individual from memory? Cluster the dataset and present the user with images representative of the mediods of the clusters. Discard images from clusters that weren't selected and re-cluster. Repeat until a handful of images remain.

Could that technique work as a virtual sketch artist and identify a person's doppelganger?