Yes, you already made that point, and I responded by saying that the States can do this and the federal government could just co-ordinate it as part of a national strategy.
In fact travel rights were actually restricted in practice by the Federal Government and it's agencies, and states under both Republican and Democrat leadership. There are many well established exceptions that restrict access to constitutional rights under various circumstances, so the contention that this can't be done falls a bit flat. If nobody can do it, how come everybody did it?
The Federal Government cannot restrict citizens from traveling between the states. I don't know anyone, here in the US anyway, who would even suggest such an action.
The privileges and immunities clause, Article IV, Section 2, Clause 1 of the United States Constitution, protects the interstate travel rights of US citizens.