Well, there is the Steam Early Access - mind you games need to be at least partly polished and quite playable before entering there. But the public is more forgiving of "early beta" quality at least there. Sometimes alpha if the game is obviously a gem waiting to be polished.
You can noodle on mechanics, art direction, story, etc with a small team before you go into production but you probably won't have an actual viable product coming out of that stage.