Downvoting on Twitter would need to have very different characteristics for the reasons you mention.
On the other hand, you actually do need users to moderate other users if you want to maintain a civil community. The alternatives are heavy moderation, which doesn't scale and introduces an institutional bias, or a free-for-all, where emotion drowns out thought, which is roughly where Twitter is today.
I absolutely agree the social graph should be leveraged. What if downvotes only by people you follow affect your feed? You already see retweets only from people you follow, so the idea that your followees influence what you see is part of the platform today. This just extends that influence into the negative direction.
Posters absolutely need to see the downvotes they are getting, the point of the feature is to moderate behavior, not just to create a filter bubble.
Obviously it's not a trivial issue and rolling out new features at Twitter's scale isn't easy either. However I'm pretty confident, with all the engineering talent they have, they could make progress if it was actually a priority.
I don't think users attempting to moderate others is a solution. It would reduce the diversity of opinions. The largest group would just push out unpopular opinions.
Instead you need tools that allows users to give themselves peace without punishing others. Maybe with the help of the followed/followers, i.e. some assistance from the social graph.