Referer is totally insecure.
If all it is is votes, I say the right solution is "let it go".
Re:Faking votes on Hacker News
by tlrobinson on Monday March 16, 2009 @09:39PM
(#519189)
AFAIK, checking the Referer header actually works for preventing CSRF because you can't modify it for the types of requests that work cross domain, i.e. loading <img>, <script>, etc tags, or posting forms.
It's also nowhere near as bad as the state of the Twitter API and apps, which require a username and password. People don't think twice about providing unlimited access to their Twitter account to random websites. Hopefully the OAuth API will fix that.
@pg: I think one solution would be to reject any vote requests with a Referrer header other than news.ycombinator.com