Note the above uses words like "possible" and "if". I find it unlikely this is what was done. More likely - like every other society - they selected for politically powerful.
Note too that it is highly likely that in the past rape of lower casts was accepted in some form, which would bring any selected IQ genes to the lower casts.
There is preliminary evidence from genetic sequencing that Indian castes have been highly endogamous for at least 3000 years.
India is a vast place with an enormous population, and there's resistance to continuing this research, for fairly obvious reasons. But continue it will, you can't un-sequence a genome after all. So we may yet see a robust answer to this sort of question.
Your final point about selection via coercion holds more water, but one has to believe that the subjects would be explicitly chosen for their intelligence, rather than for any other features, which seems difficult to support.
As I said I assume the powerful selected politically powerful mates for their girls, which might or not be intelligent as well. However it is possible that high status was high intelligence. The lower castes not following that.
Look, the above is all just barely possible. you will have to work with me here...
By segregated reproduction I just mean that most likely mate selection is done within cast, so any difference in IQ between castes is likely to be slow changing as there is little gene exchange between groups. Something similar happens in US society where there is growing stratification along IQ lines.
Looking at babies adopted into another caste, or people marrying across castes, would help shed some light. But it doesn't seem reasonable that 5k years of caste systems would have such an impact on IQ.
* What makes you think that, at any point, lower caste people had an on-average lower IQ? The same has been said for plenty of minorities that have been historically discriminated against, such as people of color and women, because opressors are in control of the narrative and see themselves as superior
* What does "segregated" (in your original comment) or "selective reproduction" even mean for you? There are more lower caste indians than total Amercians. Are americans lower IQ because of "segregated reproduction"?