Another problem is the downvoting itself. If you have a post that is liked by 50 people but disliked by 54 people, the 4 downvote majority makes it disappear, even though 50 people thought it was worthy enough to upvote.

I think the more upvotes something gets the less ability for downvotes to bury it, if the ratio is close. People always post things that are majority-liked because even something slightly disliked more than liked will bury it.

After a bit of hesitation, I find myself agreeing. At least in so far, that for posts with lots of votes it should probably be the ratio, not the difference, that governs visibility. Unfortunately, for low vote counts the opposite/current approach is more meaningful. I assume this is one of the reasons why this hasn't changed, it adds a ton of non-obvious decisions to make.

Yeah true, a system that factors in the number (ratio? total?) of votes and not just the difference might be interesting