Tell that to the US Border Patrol searching an detaining people that just happen to be near the border. It's especially fun for people that have their visa status in limbo because the State Department is processing an update. They are allowed to stay here, but all of their official papers are expired, so Border Patrol throws them in jail for a few days until they sort things out with the State Department. This all happens on regular roads that are not within distance of a border crossing.

This case is from 1973, and while it may presumably apply here, the CBP doesn't seem to think that it does.