I do the same thing, but for exactly the reasons you described, this type of passphrase can be vulnerable to social engineering.