Arguably, part of the problem is that people do. The ones who do come into conflict with the people who have no desire to use their freedom of movement.