The HSM is only the backup, it is not the primary copy. Losing the HSM means you are down to 1 copy (the vault on your device). That means there is only a small window of exposure where, if the HSM goes down and you lose your device, your passwords are gone.

They could probably roll with duplicated HSMs to mitigate this somewhat.