I'd say that if you've read and grokked Serious Cryptography, and you don't want to write code, something like _IMC_ (Silverman, above) is an interesting next step, in that you'll mostly be reasoning about math, and you'll be working at a much lower level than the basic RSA formulae.
It also depends on what aspects of cryptography are interesting to you. If you want, for instance, to get a much better grip on how bulk cryptography works, as opposed to signatures and key exchanges, your best next step might be papers. For instance, a good next step on block cipher cryptography would be the Heys tutorial: https://ioactive.com/wp-content/uploads/2015/07/ldc_tutorial...
Looks like I can't pick up IMC at my local library, so I'll have to wait a tad on that one, but I can start with papers right away. I can't quite say what my particular interests in cryptography are, but (I guess not too surprisingly) I have a desire to know more about Hash functions. Now I'm going to try to hunt down the papers for Skein. Thanks again!
Would cryptography engineering be too advanced for me? I plan on developing my programming skills after I get a computer, but until then want to continue learning.