You're arguing "you don't have a [legal/moral] right" (which is true)

They're arguing "it's a victimless crime" (which is also true)

These are related things, but they aren't contradictory