How is it speculative? The background does not always allow for the discrimination to be made but in the random sample of ~20 faces I looked it was the main factor in maybe 1/4 of the cases (I was %100 accurate on this random sample). Of course, my random sample is not your random sample. We could probably do a controlled experiment to get at these kinds of attributions systematically.

I might also say that a second major discriminating factor is skin texture, especially at boundaries.