>I mean, if the attacker has writable access to your vault, he likely can observe you handling the unencrypted password or steal all your gpg key material after you unlocked it.

To be clear here, only if that implies access to the memory you control, otherwise, no. GPG is pretty good about local file access by an attacker and pass maintains that on platforms where that is possible.