I mean civil consequences. If someone says something I dislike, and I disassociate with them, that's a consequence. Nobody should ever be jailed for their speech and that's what it's designed to protect against.

Libel is special because it's purely a civil matter and usually has to prove malicious intentions.