The actual conservative position, at least my interpretation of it, is something like, the whole point of civilization and being alive is to reach the height of human potential, build great things and make new discoveries. Cultural homogeneity and intellectual diversity are two sides of that coin. When everyone has the same basic values and assumptions in a high-trust society, it frees everyone to have the tolerance of eccentricity and the tolerance of the scariness of exploring new positions that you need in order to do anything great.

Liberals reading into this and seeing racism is itself an example of this phenomenon. They hear something like "diversity of ideas" and instantly a big red moral stop sign shows up in their head. It's an example of how when everyone has fundamentally different values, and everyone's different set of moral landmines are overlaid on the same map, it becomes basically impossible to walk through the space of ideas without stepping on one sooner or later.