I think checking referrers is the normal way to solve this. As long as your site doesn't have XSS vulnerabilities on other pages, I think you're fine, but I'm not an expert though.

EDIT: Nonces are better, ignore this reply.