What I read from authors like Jared Diamond indicates to me that this is too often couched in ethnocentric terms and was heavily dependent on nutrition. In a highly extractive, oppressive state, I could believe people might be oppressed into a lower IQ. I'd also suspect that those regions would be out-competed by regions where the average people had better diets and whose intelligence is average.

In the wilds of Papua New Guinea, lots of highlanders meeting us there would consider us clumsy and stupid and wonder how we run a complex civilization.