Given that some things that don't use words—for instance, art, money—have been ruled as being considered equivalent to speech for the purposes of First Amendment protections, I don't think the rationale you give there is likely to be the one used to justify the listed exceptions to the First Amendment.

In all the cases listed, the speech in question is being used to directly and (at least usually) intentionally harm or interfere with another person. I believe this is a case where looking to the Framers' intent rather than the strict wording of the amendment is worthwhile in determining how best to apply it. It seems obvious that they did not intend to make all forms of fraud and threats legal with no recourse (and I imagine there is some jurisprudence that cites specifics to this effect).