Well, it does matter. Why?
> Enforcing your borders and dictating the rules for anyone external crossing them to enter your country is the right of each nation to decide on its own.
Passports are part of that negotiation. An acknowledgement of the rights of the individual visiting.
As to what rights a passport holder can expect, the US is a signatory of the International Covenant on Civil and Political Rights [0]. Even if their implementation is incomplete, and heavily criticised.
However, here's some of the relevant points:
Article 9 recognises the rights to liberty and security of the person.
Article 10 requires anyone deprived of liberty to be treated with dignity and humanity.
Article 12 guarantees freedom of movement, including the right of persons to choose their residence and to leave a country. These rights apply to legal aliens as well as citizens of a state.
Article 16 requires states to recognize everyone as a person before the law.
The only reason that the US currently says they can ignore these rights and demand searching at the border without just cause, is because they say that whilst you are at a point of entry, you are neither a legal alien nor a citizen. Within the 100-mile border area, the US Constitution doesn't even apply. [1]
> Although the 100-mile border zone is not literally "Constitution-free," CBP frequently acts like it is. [1]
They actively manipulate the law to remove rights from not just aliens, but their own citizens as well.
[0] https://en.wikipedia.org/wiki/International_Covenant_on_Civi...
[1] https://www.aclu.org/other/aclu-factsheet-customs-and-border...
Note that I never said whether I personally think it's a good policy or whether I think it's morally sound. I'm saying they have legal standing to do so.
Enforcing your borders and dictating the rules for anyone external crossing them to enter your country is the right of each nation to decide on its own.