Parallel to freedom of speech is freedom of association. The consequence to exercising your right to free speech is that other people might decide you're too horrible a person to work with.

Consider this hypothetical situation. Imagine your favorite actor comes out with some really, really horrible political views, like women should be the chattel slaves of their male guardians. For a lot of people, even maybe you yourself, discovering this about a person is going to heavily color your views not only of him but of other things he does, and you're not going to enjoy seeing films he stars in any more. Since the film studios aren't idiots, he's going to be essentially blacklisted from any future roles. There's the consequences.