Also keep in mind that the Raspberry Pi Zero (used by the person you're responding to) has the same processor as the original Pi. I'm certain that the Pi 4 does tremendously better at handling crypto than the original Pi processor.

The 64-bit versus 32-bit thing is almost certainly software wonkiness, as that forum post indicates. There's no reason the 64-bit mode should be slower than 32-bit mode for that task, and benchmarks often show that 64-bit ARM is actually measurably faster than running in 32-bit mode, due to the additional available registers and whatnot.