That's so weird, I use them regularly and have never had a problem in the UK or other countries with them.

I wonder if you're failing due to not recognising your face, or (no offence intended) that you have too high a risk profile in some way due to your passport info, and they decide they want to see you in person.