I believe that in this conversation one party is saying that people have intrinsic rights (see the Universal Declaration of Human Rights) and the other party might agree on that but they say that those rights can be enforced only if they can be defended. Example: both parties probably agree that people have a right to free speech but nevertheless people end up in jail if they attempt free speech on the wrong subject in the wrong country.
You have the right to vote, if you lose that right, and you don't have a gun after that you have whatever 'rights' that are provided to you by a dictator.
One of the things you're missing here is the idea of herd immunity. While you won't fight for your rights, theoretically someone else will making taking your rights dangerous. Once enough people won't fight for their rights, or enough of the population gathers together to take your rights, you lose your rights.