My experience is different. Upvotes/downvotes are not public on HN and yet I obsess about them. I've hidden them because of that. It's not about comparing since I can't see anyone else's. I don't want to see the numbers, as it makes me think about the numbers. I rather want to see replies.
If likes, followers, etc. were private (that is, you could see who liked your post but no one else could), you wouldn't worry about what others would think of you based on those numbers. But since they're public and prominently displayed, we equate those numbers to our social value and end up with undesirable effects like comparing ourselves to others and focusing on what would get approval rather than what's authentic.