Again: I'm not an expert. But I feel I've been around the block on this topic.
Some people think it's a naughty word, but: Moderation. Or -- to put it in different terms -- a community manager. That's one way to try to enforce norms. But I'd say that the most powerful way is via intelligent growth. For example, don't just open the floodgates. Grow a community at a rate where newcomers never overwhelm the existing users. Communities that "break" can be hard to "fix" once toxic norms have been established.
The debates would probably seem fairly simplistic to today's ears, but remember that the community design of Twitter (140 chars, public, "following") was a novel idea a dozen-or-so years ago. There was a ton of experimentation with how different communities could be built. Big business at the time!
One place I've found interesting as far as learning about community design is online games. Game designers are trying to design very specific experiences for users, and you can learn a lot by looking at how Overwatch (for example) tunes both gameplay and community to craft that experience. Even when playing with a dozen strangers.
Anyway, maybe those are some leads for you.
You bring up a great point about the self policing. That's why HackerNews, ProductHunt, and many of the smaller subreddits do quite well.
But what differentiates an old fashion comment thread from a better solution if not the "technical tricks"?
Also, "in the days of yore", when community design was a hotly debated topic, what were the debates about? I'd love to learn more about that discussion.