If you say you "couldn't care less" then you don't care at all, and that's what you are trying to point out (This is the correct version of the phrase, for how it is used in every case I can think of).
Saying you "could care less", means you do care, at least a little (as Weird Al points out in the "Word Crimes" song). Again, in the places where I hear people use one of these phrases, they always mean they don't care at all, and are trying to emphasize that.
Only "I couldn't care less" makes sense for that meaning.
When the phrase you are using quite literally (hehe) means the opposite of what you mean, that's just abuse of the language. Unless, of course, you are trying to be sarcastic (which is never the case with saying "I could care less").
I feel the same way about literally/figuratively, and at least figuratively (hehe-2) roll my eyes on that one regularly.