What about letting people decide whose votes they want to count for them. You can be as meta as you want to be with this.

I don't want to count the votes of anyone my friend considers an idiot.

I don't want to count the vote of anyone whose tagged as a #foo by anyone tagged as a #bar by such and such a group.

Alternatively you want their vote to count double.

Instead of a single score you have as many scores as you have users and its impossible to interfere much in others affairs.

I think that this could be a very important improvement.