Perhaps you'd give them single-use invites, or invites that included some form of certification path? I've noticed that Discord makes me go out of my way to hand out a reusable/non-expiring invite URL instead of a 1-day use-up-to-10-times link; obviously Discord's implementation relies on their hosting all the servers, but one could imagine a cryptographic equivalent that worked in a federated protocol.

I don't use (or want to use) chat apps like Discord and such, though. It would also be a real challenge convincing the people I interact with routinely to start using such apps.

Sure; my point is that we can imagine email/phone-like systems that would work the same way.