Yes just replace "I can't" with "I rather don't want to". I don't know anybody who would imply hidden ulterior motives in this case in real life, since "I can't" is not necessarily as strictly interpreted as you are trying to do.