Unsettling existing populations increases chances of political instability and potential war. You can see the reaction to migrations and refugees (Syria, for example) now. That could quite easily be magnified if larger populations were forced to desperate measures. War is a species-ending risk, IMO.