I'm not sure how best to fix that. Some subreddits like /r/scenesfromahat use a timed-release system that only displays vote scores on comments after a fixed period (12 or 24 hours, I forget which). That at least helps reduce the first-comment effect, but it still means that anything after the votes are revealed is basically not going to be seen.
At a certain scale, I think threaded conversations are incredibly difficult to follow without a voting system to sort it out, and once you introduce a voting system you end up with voting system problems as you mentioned.
I honestly think that flat forums are more fit for purpose. Multiple concurrent conversations end up being a bit messy, but there was at least a reasonable chance that someone would reply to any given post, since everybody in the thread was on the same page.
Heck, you can even gamify engagement with a flat forum - one of those that I still frequent allows you to "react" to any given post. It doesn't actually do anything except have a number beside the post tick up, but people still use it.
Isn't that a standard subreddit setting? I don't think that actually affects the problem you were describing. I think it's only meant to help keep the vote tallys from influencing user behavior (e.g. bemoaning how many up/down votes some comment got, being extra motivated to karmawhore).
In my experience, quick-feedback scores seem to have a negative influence on people's behavior and emotional experience. IIRC, HN used to show comments scores, but stopped some years ago. I personally try to disable such "features" as much as possible.
Reddit's default "best" sorting algorithm is designed to mitigate that effect [1]. It does a good job of not biasing old comments in terms of sort order, but it doesn't help the related problem that older comments tend to accumulate the most replies so newer stuff can still get drowned out in terms of quantity of other content.
[1]: https://redditblog.com/2009/10/15/reddits-new-comment-sortin...
Instead of getting warned or banned, a user is placed in a timeout box where all of their posts get a three day lag and remain editable, and when the user logs in, they must re-read their posts before proceeding to the site. Kind of an 'in-your-face' reminder not to be a troll.
It wouldn't stop the insane trolls, but maybe it would give the borderline trolls time to think.
Kurt Tucholsky gave great advice on how to write a letter to a government agency that is applicable to all potentially heated discussions:
* write letter.
* put letter in drawer.
* wait three days. don't look at letter in drawer, write a new letter and send that one.