"I'm not comfortable with that" invites them to ask why you aren't comfortable, what they could do to make you more comfortable, etc. If you genuinely can't leave, you'll just have to change the subject or hold your ground. Don't underestimate the power of silence, which is to say blankly staring at them while they attempt to press you (if they're being plainly obnoxious).