Silence on the Wire is a fantastic book that looks at unconventional and underdiscussed aspects of security: interesting timing attacks, subtle information leakage, fingerprinting, etc. It's by Michal Zalewski, who is perhaps better known to HN folks as the author of the American Fuzzy Lop (AFL) fuzzer.

I think that book probably did more to get me into security than any other – it's just filled with really wonderful, clever stuff.