And if you had a clue, you'd know he really had no intention of being polite. ;)

Phrases like that exist to indicate that one is emotionally dissatisfied but at a level where it's bubbling underneath the surface rather than something you'd like to display fully. It gives just enough plausible deniability of the fact for it not to become the center of discussion, while still unambiguously letting the person you're communicating with know that you're pissed.