In your example, the insult is clearly directed at the third person/party. This isn't at all contradictory (or relevant) to what he's saying.