Cryptography is closer to math than most CS disciplines institutionally.

that's broadly true, but there are some areas of CS that are at least as close (say at a minimum PL theory). it's also less math heavy than e.g. complexity theory (though that's admittedly smaller). It can also be less math-heavy than learning theory. This all depends on the type of cryptography as well, especially since it can depend on the region (american cryptography is more theoretical than european cryptography, for example).