Three things. (1) User identity. (2) Moderation. (3) Storage and bandwidth. If you don't mind using different identities across subreddits, or are fine using some OIDC provider, you can replace a subreddit with something like a phpbb installation, a classical forum. Its moderation tools, while somehow different, should be adequate to thwart spam and whip rule-breakers into shape. Running somethig picture-heavy, even 1/10 of r/pic, would cost considerable money though.