It's not your problem. For the transaction to take place it has to go to your bank and your bank trusts Visa/Mastercard to manage risks. You would be surprised to know most frauds are absorbed by banks. Nobody is interested in people fearing plastic because it causes people to increase very expensive debt plus additional interchange fee from every transaction.