I suspect based on your writing you may be taking yourself a tad too seriously, and that can have huge costs in terms of anxiety.

The point being made is if it's a substantive point it doesn't matter how much the speaker identifies with it, the valid point has been made to you and it is to be considered. Hence it should not matter provided you are sticking to the facts. It seems you have made this into a personal insult about whether or not your transactional view of the costs of speaking are being respected. That is to shoot the messenger.

The person is doing you a favor by pointing out valid concerns even if they don't agree with them.