Not at all, if they take "no" for an answer and don't call back for, say, six months.

Once or twice, my answer has been "yes". In fact, that's how I got my current job.