You do have access to the upvote button,so your vote can cancel out an unfair downvote.

Comments that get downvotes also don't get upvotes, so it's not just the small group who can downvotes who are causing the comments to be gray, it's also the large group who could upvotes but who chose not to do so.