On that argument, you're advocating robbing the origin countries of the benefit of the children they invested in.
I'm not advocating anything - as it happens I do think it's unfair for some countries to cream off the best talent developed in other countries. But if those countries have failed to provide an environment their citizens can thrive in, they have every right to look elsewhere. And as a citizen of a country of migrants (*) I'm more than happy to have them come here. I recognise the UK doesn't really qualify the way Australia does but none of the concerns you've expressed make sense to me as a reason your nation's economy might be suffering.
(*) including my Dad, who came here as a 10 pound pom!
Even just having reached adulthood and attained secondary-school level education makes you a capable worker that the country they migrate to has had to do nothing to support until that point. Any small amount of extra investment to ensure they learn the language and obtain employable-skills on top of that is virtually always going to pay off.