I would place more emphasis on the downvote button causing most of the harm. If you post something, and it doesn't get up-voted, that's less of a hit than posting something and having it be actively downvoted. One is a statement that no one found it interesting enough to vote up, the other is a way of saying that kind of content isn't desired here. I'm guessing it's the downvotes that cause people to leave, leaving an echo chamber in their wake.

Upvotes can lead to karma farming, and the need to see that number of likes go up can be a problem I admit. I just think the echo chamber effect is a worse problem. We've all dealt with fragile egos and boisterous idiots before.

I miss the days where you weren't shunned for wrong-think, just argued with constantly if your opinion differed from the norm.