Yup, you actually can do it. I've seen a few enterprise level companies do it without SSO, especially for their invite user feature. Much easier to have he invite a new user to be done passwordless than the usual onboarding flow