You're trying to compress a bunch of dimensions all into one explanation of "IQ", and you're assuming a lot of things here, such as physical isolation would cause divergence in IQ; that divergence in IQ would also cause later societal hierarchical advantages (over natural resources, physical power, or other non-IQ circumstances or differences); and that later removal of the physical separation and original selection pressures would continue to select for IQ in the subgroup. Maybe once your elites get entrenched, they select for other attributes like physical ones instead. I don't see a lot of societies out there where marriages are formed based on pure intellect.