Perhaps you'd give them single-use invites, or invites that included some form of certification path? I've noticed that Discord makes me go out of my way to hand out a reusable/non-expiring invite URL instead of a 1-day use-up-to-10-times link; obviously Discord's implementation relies on their hosting all the servers, but one could imagine a cryptographic equivalent that worked in a federated protocol.