You know you have this bias, though. People use the same sort of language for racism, homophobia, and such things. You can train yourself to think of exceptions to overcome some of the bias and honestly, if you are interviewing people, you should be doing so actively. It took me longer to type up the reply than to think of valid reasons this could happen to someone, after all. As you said, it isn't rocket science.