No, Visa and Mastercard aren't common carrier and they don't need a defense. Like any other business in America they can choose not to serve you for any reason or no reason at all (so long as it isn't the illegal discrimination of a protected class)