> you'd expect it evenly distributed throughout the empire

Why? Large premodern states were almost never strongly culturally homogeneous. Especially below the upper class/elite level.