That distinction is kind of meaningless when thousands of people can enter several EU member states pretty much unregulated, then disappear under the radar and go anywhere within the EU.
That may be possible in, let's say, Belgium. Someone enters through the Italian border, somehow tricks the border patrol, and then can go with no problem from Italy to Belgium.
That's not the case for UK. On enter, you need to cross a border with officers, customs, etc. You can't even cross the border unnoticed through Ireland, as Ireland is also not a Schengen country (there's no border between Ireland and Northern Ireland, though there are sometimes checks from Police in buses and so on). There's a border also in Gibraltar...
The Brexit campaign was mainly complaining about workers from Eastern Countries like Poland or Romania, where they have the legal right to move and work in the UK, due EU accords. Plus fear that Turkey may enter in the future.
I am certain that even the most compassionate proponent of diversity would find it difficult to call the tens of thousands (?) of individuals from Roma populations in for example Romania and Bulgaria who have arrived throughout West and Northern Europe the past couple of years in any way, shape or form, "labour" ("homeless" is more accurate). I don't blame these people; I blame the governments of Romania and Bulgaria who abandoned these people, thought of them as a "problem" and decided to punt that "problem" over to other member states.
It's not "uncontrolled immigration" - an American reading this can't just turn up in the UK and start living and working there. It's free movement of labour between EU member states.
The irony, which politicians who were pro-leave are starting to hint at, is that even having left the UK will have to agree to free movement in order to have access to the EU market. People who voted leave because they were angry about internal EU migration are going to be disappointed.