Were you forced to invite them before their speech? No, your freedom of association is not affected by someone else's free speech. Burn those RSVPs all you like.

If you want to go down the everything has consequences route then there is no distinction between free speech and anything else in the universe so it's not really worth bothering with.