That's not quite true in my opinon. Take your example of academia. Many students will chose to stay in the US after (if they can, because salaries tend to be higher than in many other countries) rather than depart. Also, instead of "taking" knowledge, they might actually create some new knowledge. Think about how many brilliant professors in US universities were once foreign post-docs there. If the US chooses to close its borders to this sort of talent, it will considerably weaken itself in the longer term against countries enacting more rational immigration policies.