Collectively, we have a lot of experience with the foibles of our visual system (to the extent that we don't notice what it does poorly, which is a problem, too). Further, we've arranged civilization to play to its strengths, which is one of the reasons we're so interested in facial recognition.

The recognition systems look like they are doing the same thing we are, but they're not. Adding an adversary taking advantage of the difference is just sauce on top of the problem.