Seeing who downvoted you sounds like a good idea. Spending karma to downvote however - I see your point, but trolls or noise still needs to be punished, not sure people would still do that enough.

I don't agree with your "argument from authority" though - even pg can be wrong sometimes...