As others have said Mastodon might be what you want. You can self-host it, either "at home" or "in the cloud", and you can choose not to federate at all, or choose to federate only with instances you choose. You can choose to run as "invite only", or to allow anyone, and then eject people who behave poorly.

So you can set it up as an "invite only social network where I choose who can join", as you say.