You did make sure the AES instructions were used, right? I wouldn't be surprised the AES unit on the eMAG is relatively slow — other units seem to be as well, e.g. in my silly CRC32 benchmark, the Arm Cortex-A72 did 1kb in 79 ns at just 2.0 GHz while the eMAG did it in 103 ns at much faster clocks (3.0 or 3.3 GHz).

I suspect they might have reused these HW blocks from the old Applied Micro X-Gene :D

But now on the new product, it's all Arm Neoverse cores, it's gonna be great.