I think part of the problem is that in most cases there are far too many dimensions at work for us to understand the explanation for anything but the most trivial of cases.

Imagine there is a job to identify Fred Rogers. You train for this job for weeks, looking up imagery of Fred as a young child, young man, and as an adult throughout his career. You become very familiar with Fred's physical characteristics...his posture, his body composition, the way he carries himself, his face and any identifying features therein.

After this extensive training session you are brought in front of an audit panel and asked to explain in excruciating detail how you make positive and negative identifications of Fred Rogers. Not just demonstrate your ability repeatedly, which would just be a sampling method, but to actually describe the approach.