I think the distinction between natural rights vs legal rights comes into play here. Society has changed a lot in the time since the Constitution was written, and government isn't exactly incentivized to limit its own power in ways that reflect these changes.