That's why I always cover the pin pad with my other hand (probably also holding my wallet) when putting in a pin. However, I think the more likely scenario to defend against is shoulder surfers - the pin by itself is useless until combined with the card, so physical presence is needed to lift the card from me.