Caste systems are nearly as much an anthropological universal (at least for civilizations) as the suspicion of outsiders that the OP mentions. Across the world you can find examples of two neighbours brought up in virtually identical conditions, but with restrictions on one due to ancestry.
Racial categories mean that a white American can feel more common ground with a white Russian than his black American neighbor. American white nationalist fellow-feeling toward Russian nationalists suggests to me that it's not really about suspicion of outsiders. Racial categories cut across borders, ancestry, language, and culture.
Feeling confident that one can judge what category someone is in on sight without seeing their clothes, hearing them speak, learning their name, or knowing their ancestry is maybe not quite unique to modern racial ideas, but it certainly makes it stand out.
It's very nearly a caste system- you can be born next door, speak the same language, eat the same food, think the same way, and you will be treated differently based on racial judgements. That's quite different from "we Hill People hate the River Folk and always will!"
The modern notions of race come from a specific historical moment: the invention of the African slave trade by Western European powers. If the slave trade had not happened, ideas about race would probably be radically different. Chattel slavery on an industrial scale was a new idea and its echoes will probably always be with us.