Private property cannot hurt you, yet property ownership is restricted (e.g. you have to pay taxes). Restrictions often exist not to prevent immediate harm, but to make society functional and enable implementation of other rights. Restriction of free speech can be necessary if it is used to incite violence, thus triggering violation of other rights.