Not sure if you've done this analysis yet, but I wouldn't be surprised if downvoted posts are mostly unpopular opinions, and not necessarily incorrect facts or egregiously bad takes.
Two things are true: (1) not all downvoted comments deserve that status; (2) most downvoted comments do deserve that status. #1 sucks, but #2 is really important. The benefit of displaying #2 outweighs the cost of #1, which is not to deny that cost.
Of course it would be vastly better to only have the #2 comments displaying as downvoted, but with a community mechanism like downvoting, there's unfortunately no way to have only the benefit and not the cost. Moderation can help, though (we'll often restore comments that are unfairly downvoted) and maybe there's a place for something like vouching to return a comment to a neutral state.
I'm not sure if -1 points is really a great signal. That could mean only two people disliked your post, and that kind of small buffer greatly disadvantages unpopular opinions. I would think a post that is really problematic would probably amass at least net 10 downvotes.
Do you take volume of total votes into consideration, e.g., do you treat a post with 2 downvotes the same as a post with 10 upvotes and 12 downvotes? I would think a post with a sizable number of upvotes would be more likely to be an unpopular opinion rather than a really problematic post. Maybe the threshold for greying could be lowered for posts with a sizable number of upvotes.
I will look at this more closely though, because it would be interesting if there were a noticeable difference: "do you treat a post with 2 downvotes the same as a post with 10 upvotes and 12 downvotes". Thanks!