I think it's probably related to the fact that, for most people, there's a right amount of selfish and that amount isn't anywhere near 0. People would much rather get a no than a reluctant yes, even if they themselves don't realise this.