In fact, if you want actual free trade, then trade in labour must also be freely allowed.

Freedom of movement to everyone, as inside the EU, would make even more sense. Every country is rather protective of that though.