In the US, at least, the underwear drawer is a popular place to hide other things ... eg, your illegal drug stash, or the engagement ring you bought, etc. I think there's also subtextual implications of nosing about in your sex life.