I would suggest that if someone "whines" 'you mean figuratively, not literally' that they may be double-checking (depends on context of course). Or perhaps the complaint may be due to the fact that the complainer does not know that "literally" can mean "figuratively."

I agreed that the word "literally" can be used figuratively, but this use needs to be obvious- kind of like pronouns. I personally prefer to avoid that use because I am not a particularly talented wordsmith.