It may not be said eloquently or politely, but assuming you know better than people because you don't like the language they use sounds elitist to me.

That said, choosing who you do and don't interact with is your own business. Perhaps "I know I know more than them because they use homophobic slurs" isn't the strongest reason though, as one does not always map to the other