Our rights have been alienated when the government felt like it. Which is true for all supposedly unalienable rights.

You have rights as long as you are are a recognized member of the political system that controls the territory where you live. But there are plenty of people "on the margins" who simply don't have rights and governments reserve the right to expel people from the political system at will.