Convolutional layers are designed, by and large, and they're mostly the same everywhere. Yann Le Cun came up with them in the mid-90's, but their academic origins go back to at least the 50's and 60's.

then the results of this paper are sort of surprising that it's that good, if the structure is just relatively old, hand-crafted and traditional.

Handcrafted, but auto optimizing is a hot research topic right now with DeepMind et al. Needs to be evolved at deeper level than what they’re doing now so architecture is discovered, not just optimized.