Interesting! This seems to be the crux of the disagreement. What thinkers are you referencing? I'd love to read them.

At first blush, my thought is that you could still create a secular argument for natural rights, but I'm open to being persuaded otherwise.