If they say stuff like that, they aren't friends.

The polite response is something like: "I hear that as shaming, and it offends me."