It's not like we would have free roaming in the modern age if passports weren't a thing. People would still face travel restrictions, they'd just be more nonstandard and unpredictable.

Consider how the days before passports also lacked air travel available to the masses. Even cars has only just started to become commonplace. I don't see how you could possibly have travel that is so cheap, quick, and accessible to such a large population without some way to control who is coming and leaving your country? Especially with how we're quickly making different parts of the planet inhospitable, and given how rapidly populations have risen compared to a century ago.