It's a very interesting idea to have users see only the votes of other users they choose. One problem though is that it could easily lead to leaking who voted for what, which would make a lot of users (including me) uncomfortable. E.g. your pool is your 7 coworkers, and user x. One day you know all your coworkers are on a plane, and that you're therefore seeing user x's votes. Another problem is that it could be a lot more expensive to generate pages.
My original thinking was to suggest a combination of enforcing a minimum number of trusted users and then reporting a weighted average of your trusted users + the mass' rating but your solution is conceptually nicer since it adds the same noise to the vote but while extending the idea of valuing a particular user's judgement.
The big technical issue will be computing these "trusted ratings" for every comment. The trust graph could be done asynchronously so that wouldn't be an issue.
The only way to fix this I can see is either some form of meta-moderation where you hold people accountable for their votes (ala slashdot), or the system I'd personally rather see, where rather than a full-democracy the users pick whose votes they see on comments. I'd have my pool of 20-30 users whom I trust and I could see their scores vs the throng's.
Just as you hand-picked some users as moderators who you feel represent what you want HN to be, let us pick who we want to serve as our content filters.