"The UK was one of the few net contributors to the EU" -> UK was in very good company as can be found easily here: https://www.statista.com/chart/18794/net-contributors-to-eu-... and it's not difficult to see that the amount of the net contribution is somewhat proportional to the share of the European GDP .

"They didn't want unspecialised poor immigrants from Eastern/Southern Europe" -> the fact is that they were getting a lot of graduated/high specialized immigrants from southern Europe as well.