I don't think we're there, yet.

Sadly, I begin to conclude that right now most downvotes mean: "I really, really hope that isn't true, even though you have a bibliography and I just have an emotional reaction."

Maybe a Markov-chain based reputation system (a la Google for search) that makes downvotes unequal is in order.