Which is why I said "..HTTPS is a good idea".
There are valid reasons why serving everything under HTTPS isn't always a good idea. The obvious one is CPU cost, but cache performance can also be affected. See, for example: http://blogs.msdn.com/ieinternals/archive/2010/04/21/Interne...
http://blog.pluron.com/2008/07/why-you-should.html
(I'm not saying that https isn't the best option. I'm just pointing out other options that can work with plain http.)
https will swallow referer automatically.