I'm pretty forgiving -- heck it usually takes at least half the interview before the candidate's nerves settle down.
Being a developer involves being able to explain yourself clearly. You can't just sit in the dark and code, talking to no one. If I came away from an interview with the impression that you can't code then I don't want you. The candidate has to do some work too -- they are there to convince me to hire them, not just pass a test and get a grade.
If you can pass a coding test but you can't express yourself in a way that anyone can understand, then that's a no-hire. If that's a cultural bias, so be it. I don't see how it can be any other way.
Now it's been my personal experience that culture doesn't matter too much. For more than half the team, English is a second language. HR will reject candidates for language reasons even before I get to them.
How do you know? Do you check your interview assessments against actual performance among those hired (measured how)? What about the ones you turned down, what makes you so confident that they actually couldn't program as opposed to just answering differently from how you expected?