Crossing a legal border physically doesn't give you the right to be there, and work, receive benefits, etc.
So anti-Immigration Brexiteers don't mind illegal aliens?
I doubt that.
A border will do nothing against illegal EU immigration. I doubt any brexit scenario will result in EU nationals requiring a visa to enter the UK (for short term stays). A border is not what will prevent them from overstaying.
So, we're going to ruin the entire country to get immigration controls (and destroy trade, and cut ourselves off politically, socially, and from the scientific communities, and...) then just leave the border open?
That's like burning down your luxury rental accommodation because you don't like that exact shade of wallpaper, then moving in to a hovel with graffiti on the walls.