There's no need for a visual desktop just to enter the passphrase. You can also do it by logging into a minimal setup that is inside the initrd via SSH (dropbear). It is a standard debian feature.