The absurd part is you are acting like everyone who uses that phrase has casual access to the interior life of absolute strangers: if it isnt a myth and the only reason someone wouldn't point it out is politeness the it follows that you know their motivations.
For example, many thinks that saying "I have many black friends!" is just virtue signaling by someone not very virtuous, and that is often called out. The virtue there is to have those friends, ie not being racist about who you choose to be friends with, the signaling is that you say you have black friends.
See the difference? And I bet you think that someone saying "I have many black friends" is more about signaling than being a virtuous act in itself. Many people signal that way, words are cheap actions are not.
What I don't get is why nobody says "stop virtue signaling" to those people, because that is what they are doing. Virtue signaling is absolutely a real thing and people do it everywhere, regardless of political alignment etc.
Of course, people can try to smear genuine virtue by calling it mere virtue signalling. Philanthropy might be called mere virtue signalling by some, even though it does materially benefit people. But I think most cases are more clear-cut.
Ah ha! The misunderstanding is on your end!