My response to that is the same as it has historically been. We don't need a wall to keep our people in. If you wish to go to North Korea, you are free to do so.

Fair warning though, it doesn't work the same way in reverse.