unsync your contacts from whatever service provider you're using, make sure they're gone, go ahead and share the contacts (which are now empty) with Clubhouse, get the invites, then revert everything back?