To add to this good explanation, that's also why slander and libel are not protected forms of speech (but they do have high standards of evidence).

You can call a person a million names to their face, but if you publish an article or book saying you think that person's a thief, and that person's life was ruined because nobody would hire them anymore, then yes you're potentially opening yourself up to legal action.