Exactly. More generally, no one's rights are absolute, because eventually exercising your rights begins to infringe unduly on others'.

It also makes good sense to me that certain types of speech, such as harassment, result in less freedom overall. (In the case of harassment, because it suppresses speech from the harassed party.)