Well there are certainly many freedoms that we restrict or completely deny just because of the potential to misuse them (e.g. freedom to own a gun).

I don't think this is the case though, and I agree with all of your second paragraph. But I do think (and it seems that) potential to misuse a freedom is sometimes enough to restrict it, even completely.