If someone says that the phrase 'some people are smarter than others' is racist, that's specifically them being racist, (as they're the ones who brought race into it in their own mind) not the speaker of the phrase.