> The worst part is that when I call the bank to see if its legit, they are much less pleasant to deal with than the scammers...

+1

This is so true. I just never realized that is why I'm always tempted to not bother doing the right thing.

I've stopped trying to call - if I think there's a problem I go into my local branch. Much harder to put me on hold for 40 minutes and then hang up in person.

I just realized that's an excellent opportunity for "reverse phishing:" you can mangle the first 4 digits of your card [or make one up wholesale] and if they say "thank you, sir" you know they are fake. The real bank will spot that mistake instantly since that prefix is per financial institution

They know about bank prefixes and will ask you to check the number again.