The reason why the right to not be offended is a positive right is that I am obliged to find out what might offend you, which is impossible when speaking to a large crowd, and if there is a need to express something along the same ideas I am forced to express it as you would or not at all. Eg if you believe everything is a social construct I can’t express what I think is influenced by biology if it offends your beliefs.

However, you are correct that the line can be blurry and right in general should be granted with great caution.