I'd argue that if you don't regularly dust it off, you'll forget it. The only reason I remember most of that stuff is because from time to time, I go back and read crypto papers, and then need to refresh my memory.

Think about any foreign languages you learned in high school. If you have never used them in 10+ years, it is very unlikely you'll be able to 'dust off' your second language as languages acquired late tend to require regular use before amnesia sets in.