My personal impression is that you're right: there's quite a lot of women cryptographers (thinking of "attendees of crypto conferences", but of "top figures in the field"), but these women are disproportionally from a mathematics background.