This is actually a serious debate in the political philosophy community, notably with georgists.
Travel implies that you are traversing some real property on land, sea or air. So the ownership (or lack of ownership) of the property that you're traversing would in theory dictate whether you could cross it and how.
Different countries have different laws in this regard, including "easement" requirements, but it's not universal, hence national borders.
So while it seems like you have the "right," it's not so simple legally or philosophically to determine who is guaranteeing or violating that right at any given moment.
You are so fixated on the car in this mindeset.
Do you have the right to fly an airplane and have privacy at the same time? If you don't have it in the air, why do you think you should have it on the roads?
This same style of argument when applied to freedom of speech leads to the appropriately criticized "free speech zones". I have a right to travel and a right to privacy. Saying that I can only use one of these rights at a time is absurd. My opinion on this specific instance might change if the US invested as much into public transportation as it does into roadways, but that is not the case.
> I can't really think of a situation where I would care that someone is passively recording my plate location as I drive by?
Driving to a union meeting? Driving to an abortion clinic? Driving to a church of a different religion/denomination than your family? Driving away from an abusive partner?