One thing that would solve the problem with growth would be partitioning the "community" into smaller groups that are manageable by each user. Reddit does it through subreddits and it works somewhat. I would favor an approach where users could white/gray/blacklist other users, and use these as modifiers to the overall score.