Something to consider, sure.
But the exposure is to anyone with access to your encrypted pass data. Which in the normal use case is going to be anyone with access to your user account, which means they could likely already see your shell and browser history.
pass uses normal folders to store your website/username information so in that way it is less protected.