You bring up a great point about the self policing. That's why HackerNews, ProductHunt, and many of the smaller subreddits do quite well.
But what differentiates an old fashion comment thread from a better solution if not the "technical tricks"?
Also, "in the days of yore", when community design was a hotly debated topic, what were the debates about? I'd love to learn more about that discussion.
Again: I'm not an expert. But I feel I've been around the block on this topic.
Some people think it's a naughty word, but: Moderation. Or -- to put it in different terms -- a community manager. That's one way to try to enforce norms. But I'd say that the most powerful way is via intelligent growth. For example, don't just open the floodgates. Grow a community at a rate where newcomers never overwhelm the existing users. Communities that "break" can be hard to "fix" once toxic norms have been established.
The debates would probably seem fairly simplistic to today's ears, but remember that the community design of Twitter (140 chars, public, "following") was a novel idea a dozen-or-so years ago. There was a ton of experimentation with how different communities could be built. Big business at the time!
One place I've found interesting as far as learning about community design is online games. Game designers are trying to design very specific experiences for users, and you can learn a lot by looking at how Overwatch (for example) tunes both gameplay and community to craft that experience. Even when playing with a dozen strangers.
Anyway, maybe those are some leads for you.
But there's also just non-technical stuff. Which I would argue is actually more important, since any piece of social software can be perverted if a certain subset of users is determined. A non-technical example: setting community norms. Hacker News also does this reasonably well -- the community doesn't respond well to goofy memes, for example. This isn't something you read in the rule book, it's something you experience by reading comments and feeling out the tone of the conversation. And if you do post something that doesn't fit the community, you'll pick up a few downvotes or some-such. Most users understand the nature of the community so it self-polices relatively well (with a little help from moderators). I think large open communities (Twitter, Reddit) that need to maintain a ton of traffic to make money have trouble with this: It's tough to want to service the community needs of the entire planet and tell people "Behavior X isn't something that's appropriate here."
I'm not an expert and before you tear me a new one, remember that this is a Hacker News comment and not a peer reviewed journal article. I'm just tossing out some thoughts.
But my belief is that designing for communities is actually quite a challenge, and the fact of the matter is most people putting communities up online right now don't really put much thought into it. Which is a shame. It allows a small subset of toxic users to really damage what should be a good thing for the rest of us.
Back in the days of yore -- late 90s through 2006-ish -- community design was actually a pretty hotly debated topic. But, like I said, it really seems like these days old fashioned comment threads (like you see on news articles), Facebook, Reddit, and Twitter have established themselves as the blueprint for community design and people just sort of copy those without thinking about how they do and don't work. And then online communities become toxic and some people assume all online communities are toxic and then some people start thinking all people are toxic and -- hey! Look where we are. (I'm stretching a bit at the end, here, of course. ;-) )