If you study, you may find the right to travel was already litigated at USSC. The gov't will not defend that right when most everyone signs it away by voluntarily applying for a license which has rules and fees. Gov't funded schools obviously don't teach this.

For extra credit, research what requires you to have an SSN. Hint - SSN is not part of passport application. Spoiler - your parents sold you out for the IRS tax deduction.

I suspect you could, if you only used roads built with your own money, on land that you own.

That's not what USSC said. You could look it up and find the driver license for non-commercial use is voluntary. Or keep suspecting things based on how things seem to be and fictional theories.