One of the theory on face recognition is that we learn to recognized people by "storing" the difference between an average face and the people we are looking at. And we build that average face from the sum of all people we met.

https://en.wikipedia.org/wiki/Cross-race_effect

So indeed the different training set has a big impact.