If not wanting to live in an enclave as a nominally second-class citizen is racist, then I don't know what to say. Let's not deny reality here, that's really what you end up being if you try to interact with the main racial group in a rural area, or at least the vast majority of them. It doesn't matter what race this happens to be, it's how people work.

I know an Englishman who lives in an Indonesian village (not an enclave). He is constantly reminded that while his presence is cherished, he isn't, and never will be "one of them." He still loves it, though, so there's that.