I'm worried about that too, but you can deal with it by making the system that automates your backups also test your memory of the password (while still creating a non-interactive backup even if you ignore the prompt). Alternatively, reuse a password that you regularly use offline, like a disk encryption or SSH passphrase.
> If we perform similar backups to the same remote destination over SSH (Borg) and SFTP (Restic), the initial backups take roughly the same time, but the subsequent incremental backups take something like 10x longer with Restic.
Also I'm wary of encrypting personal backups. I think the chances of me forgetting a password that I used literally once are quite high, and what's the point of a backup if you can't actually access it?