One reason why the PI can be slow here is lack of any ARMv8 Crypto Extensions and by running in 64 bit mode it can make it even slower.

https://www.raspberrypi.org/forums/viewtopic.php?p=1705614#p...

Also keep in mind that the Raspberry Pi Zero (used by the person you're responding to) has the same processor as the original Pi. I'm certain that the Pi 4 does tremendously better at handling crypto than the original Pi processor.

The 64-bit versus 32-bit thing is almost certainly software wonkiness, as that forum post indicates. There's no reason the 64-bit mode should be slower than 32-bit mode for that task, and benchmarks often show that 64-bit ARM is actually measurably faster than running in 32-bit mode, due to the additional available registers and whatnot.

Are you using RSA or ECDSA certificates? RSA signing is a lot more difficult than ECDSA signing. If you're concerned about compatability with older clients, you can have dual certificates, but that can be more time consuming to setup.